IET Gen Development Blog

Static Code Analysis

2012-01-24T15:32:00.001Z

Of the 150 checks currently available in VerifIEr, a good many are aimed at detecting errors in code. If you can find an error in the code prior to code generation, it is much easier and quicker to fix, and fixes an error that might not get detected during testing.

Until recently my favourite check was one that detects views that are used but not populated, since this usually indicates an error.

My new favourite is our latest check, which looks for ambiguous OR clauses in an expression. For example, the following code is ambiguous: IF x=1 OR x=2 AND y=3 and should be written as IF (x=1 OR x=2) AND y=3.

We ran this check on our current code base and a few clauses were highlighted. As an example of how difficult it can be to spot these errors, can anyone see the error in this READ statement? We didn't, but the check did!

Unadopting action blocks

2012-01-05T14:28:00.001Z

A customer recently contacted us with an issue related to having two action blocks (in different models) that had the same ancestry but were otherwise different, i.e. different names, logic, source code member name, etc. Because they had the same ancestry, GuardIEn was managing them as the same deliverable, but they needed to be managed separately.

The solution was to give the second copy of the action block different ancestry, but how, since there isn't an unadopt function on the CSE.

The workaround was as follows:

scope a subset containing the ABs that you want to unadopt
download the subset
generate a new (temp) model on the CSE from the subset so that the ABs in the new model have new ancestry
selectively adopt just the business systems in the new model to the old model
selectively adopt the ABs in the old model to the ABs in the new model so that the ancestry in the old model changes
delete the new (temp) model

View Starving or Perfect View Matching?

2011-12-08T08:51:00.001Z

A recent question from a customer was related to view starving and perfect view matching.

The technique of view starving involves removing all redundant attributes from views to avoid unnecessary storage allocation and view initialisation logic.

The technique of perfect view matching involves ensuring that the view structures in a USE statement are identical so that the program call can be achieved via passing of the actual views.

The question related to the problem that if you starve the views, you might not get perfect view matching, and hence should you then allocate a new local view that has perfect view matching and add in extra statements to manually maintain this temporary view to achieve perfect view matching for the USE statement.

If you do not have perfect view matching, then Gen needs to generate an intermediate data structure in the calling program to accomplish the parameter passing on the call of the used action block.

Data needs to then be passed from the actual views in the calling AB to the intermediate data structure and therefore there is an overhead for a) allocating the storage for the intermediate structures and b) the instructions to move the data.

Therefore in this situation, there is little difference between doing this yourself via a local view or getting Gen to automatically generate the extra structures – it amounts to the same thing, at least for one USE statement. Note that the order of the views is also important. Your local view would have to have the same attributes in the same order, and to work this out requires a careful comparison of the two views since the view matching dialog will not indicate if they are ordered differently.

It only becomes more efficient to define your own views if you have many USE statements that would make use of your fully populated view, since Gen would generate the data moves for each USE whereas you may only need to move the data once for multiple calls. However this becomes more complicated to understand and maintain. It only needs one attribute to be added, deleted or even moved in sequence for the technique to not accomplish perfect view matching, and then you get a double overhead – your views and the extra generated code. The additional code also complicates the action diagram, often for little benefit.

In an on-line transaction with a single USE statement, the overhead is not worth worrying about. You should concentrate on achieving perfect view matching where it will affect performance. This would typically be for large group views and repeated calls to the same action block, for example, in batch jobs where the same action block is called within a loop that is executed many times.

With our automated code checking tool VerifIEr, we have a perfect view matching check. This will work out whether you have perfect view matching or not, and can be configured to only check group views and USEs within loops, so that you only have to focus on the important ones. VerifIEr can also check that attributes are used, and hence help with view starving.

Where you want to have perfect view matching for performance reasons, I would add in the extra attributes to the ‘proper’ views rather than add in extra views, since there would be no benefit from adding in the extra views. The only time this might be needed is if there are many called ABs with different import views and all sourced from the same view in the caller. However in this situation I would then let Gen handle this situation rather than introduce my own additional views.

Trace Lite

2011-11-18T16:38:00.000Z

When you can't work out what is going wrong with your code, you can often solve the issue by stepping through the statements using the Diagram Trace Utility (or xTrace on the z/OS platform). However this requires regeneration of the code with the trace option enabled and this takes a bit of time.

In many cases, just seeing which statements were (or weren't) executed is enough to locate the cause of the problem. You can now do this if you are using pathvIEw, since it has a feature to display the last executed statements.

Moving beyond the 32k CFB limit

2011-10-25T12:31:00.004+01:00

Gen r8 IE2 increases the Common Format Buffer (CFB) limit from 32k to 16M.

The increase in the CFB limit is available for C generated applications on Windows and UNIX, but not in this release for z/OS, and hence we will not be able to take advantage of this for our products until z/OS support is available for CICS and IMS COBOL servers. It must be the top ranked enhancement request for CA Gen for the past 20+ years, so it is great to see this finally make it into the product.

Previous posts have discussed strategies for coping with the 32k view size limit, and our experiences with a Web View interface has been that it is a good idea for web applications to fetch and display small pages of data rather than trying to bring back a huge results set into a group view. The 32k limit therefore can be a good thing because it provides a limit on the amount of data returned in a single server call.

With the new limit, a developer could massively increase the data returned by a server to ~16M, which might be a good thing if the application did this anyway with repeated server calls, but a bad thing if the user was previously responsible for paging through the data and using filter/selection fields to limit the number of rows displayed, but they can now display thousands of rows without needing to worry about the filters.

Thinking about the impact on our own products (which are developed with Gen), there are several servers that would be much simpler with a larger export view size, so once z/OS support is available, we will take advantage of the larger view sizes to simplify the code and improve performance.

Gen 8 Interim Enhancement 2

2011-10-25T12:31:00.000+01:00

We have just started beta testing Gen r8 Interim Enhancement 2 (IE2).

(Interim Enhancement is the new term within CA for Feature Pack).

The new features that we are particularly interested in are:

Increase in the common format buffer (CFB) limit from 32k to 16M
64-bit Windows applications
Customised Java proxies

Visualising Data

2011-08-30T15:40:00.000+01:00

With the development of pathvIEw, we wanted to provide a graphical display of how complete the testing was for each module. To do this, we developed a new function in IETeGUI to provide a percentage complete indicator to display in a listbox cell, like this:

This got us thinking about other ways of visualising data to make it easy to glance at a table of data and see the most important bits. We therefore created two additional bar chart styles which are illustrated below. The first style is just a single bar and is used to illustrate the number of statements - the longer the bar, the more statements there are.

The second style is similar to the first, except that the colour is based on a medium and high threshold. We have used this for the complexity column, with green showing low complexity values, amber for medium complexity and red for high complexity.

The nice part about using IETeGUI is that the above is handled without the need for any additional OCX controls and is displayed in a standard Gen listbox.

Cyclomatic Complexity and CA Gen

2011-08-26T14:45:00.000+01:00

As part of the development of pathvIEw, we have introduced a new complexity metric which is calculated for each action block and procedure step.

We conducted some research into the usefulness of the metric and this shows some interesting results into the use of complexity counts for predicting error rates in code and hence for its use in prioritising testing efforts.

See the research document here: Cyclomatic Complexity and CA Gen.

Detecting Redundant Code

2011-08-25T12:36:00.000+01:00

We recently launched pathvIEw - a new Code Coverage testing tool for Gen, and since using a product yourself is the best way of gathering ideas for enhancements and improvements (as well as testing it of course), we have been using pathvIEw as part of our testing to ensure that we have covered all of the paths through the logic.

When you find that some branch of the code has not been executed during the testing, this could be because your testing has not been thorough, but it could also mean that you cannot get to these statements, i.e. it is code that can never be executed and hence can be removed.

Whilst testing some new functions in pathvIEw recently, there was a block of code that had never been executed, and when I looked at the path through the logic to this code, it became apparent that it was never going to be executed, and hence could be removed.

Web View Experiences - View Sizes

2011-07-29T16:49:00.001+01:00

Our GUI client design for list boxes is to retrieve all of the rows that satisfy the filter criteria and then allow the user to easily scroll up and down through the list box. Because of the 32k view size limit on the server procedure step, the client list box size is much larger than the server view size. For example, the client list box might allow up to 32000 rows (the limit for a GUI list box is 32767 rows) but the server group view size might be 100 rows, so the client would call the server multiple times to retrieve all of the data.

Whilst this approach works well for a GUI client, it presents performance problems for the web interface. Therefore our design for the web interface is to only call the server once to retrieve the first set of rows (for example 100) and then provide Prev and Next buttons on the web page to support paging.

The servers will already support the Next page operation because this would use a 'start from' field that would be set in the GUI client to page forward through the data. However our servers typically did not need to support a 'previous page' operation and so this has had to be added to the servers to support the web interface.

Web View Experiences - Introduction

2011-07-29T09:57:00.001+01:00

In the latest release of GuardIEn (8.1) that we launched this month, we introduced a web based user interface for viewing and managing Change Requests and Release Packs. Called WebCR, the primary purpose for developing this new interface was to provide an easy way of giving a wider group of users access to the change management functions of GuardIEn without needing to install the full client software on their workstation.

All of our products are developed with CA Gen, and we also wanted to develop the web interface using Gen so that we could maintain all of our code in a single tool, and share common logic between the GUI client and web interfaces.

In the past we had experimented with Gen's Web Generation capabilities, but decided that these did not meet our requirements. However with the introduction of Web View in Gen 8.0, we decided to re-evaluate Gen's capabilities and started an R&D project with Web View. The results were promising and the decision was taken to proceed with the full scale development of a new web based user interface for the change management aspects of GuardIEn.

The design criteria was to provide similar functionality as the GUI client and use the same servers as the GUI, so that a customer wanting to use WebCR simply needs to install the WebCR application in an application server and the user interface is up and running.

Whilst we wanted to provide similar functionality to the GUI interface, our research project indicated that there were some important differences between GUI clients and Web clients with respect to dialog design, window layout and performance. This meant that we needed to have separate client procedure steps for the web interface, although they could share the same servers as the GUI client and also common code.

In subsequent posts, I will describe some of our experiences of using Web View in more detail.

RI Triggers – Gen or DBMS?

2011-07-08T12:49:00.000+01:00

Sorry for the delay since the last posting, but we have been very busy at IET with launching a new release along with a couple of new products.

We recently had a discussion with a customer regarding the difference between Gen and DBMS RI enforcement.

The advantage of using DBMS RI is that the RI integrity is maintained by the database and not Gen generated code and hence any program or interactive SQL that deletes records will ensure RI integrity is maintained, whereas with Gen RI, you must either always perform deletes using Gen programs or ensure that your non-Gen programs or SQL correctly maintain RI integrity by cascade deleting child rows, setting foreign keys to NULL, etc.

However one important consideration is that many DBMS products do not support the full range of delete rules that can be defined in Gen. One example is a pendant delete, where the parent row is deleted when the last child is deleted. In this situation, Gen will enforce the rules that cannot be enforced by the DBMS, so that the generated RI triggers contain a mixture of Gen and DBMS enforced rules.

The danger with this situation is that you might think that all RI is enforced by the DBMS and hence not worry about deletes performed outside of Gen, however the DBMS would only be performing some of the deletes and hence the results would differ between using Gen to perform a delete compared with non-Gen programs.

Another consideration with DBMS RI is that you must ensure that the DBMS rules are kept up to date, on all databases, for example, development, test and production.

For these reasons, we use Gen enforced RI for our products.

GuardIEn and Versions

2011-02-23T08:40:00.002Z

I was recently asked to clarify how GuardIEn handles code versions, and thought that posting the reply might be helpful to others.

GuardIEn stores meta-data about versions (i.e. information about the versions) rather than the versions themselves, and the actual versions of objects have to be stored as objects in Gen models. Hence to be able to ‘use’ any prior version in a Gen process like a migrate, it must exist in a model.

If you consider an example of a 3-level development hierarchy with DEV, TEST and PROD environments, then as long as you have a Gen model associated to each environment, you could have up to 3 separate versions of an object, one per model.

If you start with the same copy of the object in each and assign it Version 1, then in GuardIEn there will be one version (V1) and that version is in the DEV, TEST & PROD models.
You change the object in DEV and upload it using the Upload Assistant, GuardIEn will create a new version (V2) and assign it to the first state in the life-cycle for the DEV environment. You then have two versions in GuardIEn and two versions in the Gen encyclopaedia – V1 in TEST & PROD and V2 in DEV.
You migrate the object from DEV to TEST and the status of V2 will be updated to a TEST state and you still have two versions, V1 in PROD and V2 in DEV & TEST.

If you want to backout the change in TEST, you could migrate the object from PROD to TEST (assuming that there were no other changes in the model that would prevent this migration) an reset the status of the version to a DEV state.

Once V2 is migrated to the PROD model, you no longer have a copy of V1 in any model and could not go back to V1 through migration.

It is often useful to be able to see what the contents of previous versions was even if you do not have them in a model, and to support this, we have support for Minor Versions. Whenever an object is changed via Upload Assistant or genIE, a copy of the object is taken as a text file (PAD listing for AB/PROC, objects & properties report for ENT/WAS, source code listing for XOs) and stored in the GuardIEn database. This provides a full audit trail of each change made to an object. Whereas in a Gen model you only see the latest copy of an object and hence the detailed who/what/when of multiple changes to the same object are lost in Gen, with Minor Versions you would see all of the changes separately.

This is then a very useful resource, not only as an audit trail of changes, but also for problem solving. The ability to see the what, why, when and who (what has changed, why was it changed, when was the change made and who made it) makes diagnosing a problem much easier. With Gen, a single model can only contain a single version of an object, so if the object is changed, you lose the ability to see what it looked like the moment before the change, unless you have saved the previous version somehow (via migration, model copy, etc.).

It is possible to configure GuardIEn to manage a backup model which is maintained as part of a system update to production. The section on Backup Migration in the System Updating Steps manual describes the process that the backup migrate step uses to scope the objects that are migrated to the backup model. However because a Gen model enforces strict consistency between objects, it may not be possible for the n-1 version of two objects to co-exist in the same model, for example, if the n-1 action block uses a new attribute from the version n entity type. Hence often you may find that the current production version of an object has had to be migrated to the backup model to support the migration of another object and the backup model does not therefore contain the previous version of the object.

Gen 8.0 FP1 and Batch z/OS Libraries

2011-01-17T14:57:00.004Z

A previous posting described the new Gen 8.0 feature for z/OS dynamic RI triggers and libraries and an issue related to batch support. This has been addressed in Gen 8.0 Feature Pack 1 which is currently under beta test. FP1 also introduces some a useful enhancement to host encyclopaedia construction in allowing you to specify separate on-line and batch load libraries, so that when a dynamic action block needs to be linked separately for on-line and batch, this can be done from the same business system into separate load libraries.

Continuous Integration and Gen

2010-12-03T16:59:00.001Z

Continuous Integration (CI) is a practice often undertaken as part of agile development or extreme programming approaches where development team members frequently ‘integrate’ their work, often every few hours.

The requirement for code ‘integration’ most often results from the ability in most development approaches of allowing multiple developers to checkout the same source code and then rely on then ‘integrating’ their changes back into the master copy stored in a repository. It also typically results in an automated build process (and possibly also automated test) to ensure that the integrated changes are compatible with changes applied by other developers to the same or related source.

The objectives of CI are to improve the quality of software and reduce the time taken to deliver, especially by reducing or eliminating the costly and time-consuming integration tasks by ensuring that any integration issues are resolved early on and not as a massive exercise late on the development process.

Some of the technical issues that CI attempts to address are not applicable in a Gen environment. Only one person can checkout an object with modify access from a model and on upload, Gen ensures that at a basic level, the changes are consistent with the model, and so it is tempting to conclude that CI is not applicable in a Gen project.

However there are still several areas where CI concepts can be usefully applied.
The first relates to generated code and test environments. In our development environment, we generate code from the CSE into a shared development directory. In this way, every developer tests from the same code base and does not need to worry about maintaining their own private source code, object code or database. There are several other benefits of using server based code generation over local toolset generation, and perhaps this could be the subject of another post sometime…

It is important that the changes to the model, once uploaded, are correctly generated. We therefore use GuardIEn’s Upload Assistant to automatically perform the impact analysis and then generate the affected modules after each upload. In this way, the generate/build process ensures that the development code repository is kept up to date, and any errors are trapped at an early stage.

Another aspect of CI is ensuring that quality control is applied continuously. Numerous studies have shown that errors are far cheaper to fix if they are detected and fixed at an early stage in the life-cycle. We run about 25 checks on the objects changed automatically on upload using the integration between VerifIEr and the Upload Assistant. These checks detect common errors in the code (for example missing view matching or redundant code) and whilst the errors should be detected during testing, it is far easier and cheaper to correct the errors whilst the subset is still downloaded and before time has been wasted generating and testing the code.

Alternative View Mapping Technique

2010-11-08T13:52:00.005Z

The recommended technique for passing data between action blocks is to use view matching on the USE statement. One disadvantage of this method is that the views need to be mapped all the way down the calling chain. For example, if we want to pass some data from AB1 to AB9, then the view(s) must be mapped on every possible USE statement:

AB1
->AB2
-->AB3
--->AB4
---->AB9
->AB5
-->AB6
--->AB7
---->AB8
------>AB9

With some very complex structures involving hundreds of possible paths through the logic, this can involve a lot of extra views being created in the intermediate action blocks in the calling chain and the potential for not mapping some of the views, thus the data is lost during the calling chain.

A technique that we have used to provide an alternative method of passing data around is to have a common action block that stores the data in an uninitialised local view.

The logic of the action block is as follows:

SAVE_DATA
IMPORTS
in action code
link my_data string (exported)
LOCALS
temp my_data string (not initialised)

IF in action code = 'P'
MOVE link my_data to temp my_data
ELSE
MOVE temp my_data to link my_data

The revised logic for the application is now:

AB1:
SET temp action code to 'P'
SET temp my_data string to 'whatever data you want to pass'
USE SAVE_DATA
WHICH IMPORTS: temp action, temp my_string

Any action block that wants the value of my_data can then use SAVE_DATA to get the value without it needing to be passed on every intermediate USE statement.

Note that this technique will only work within a single load module and cannot be used to share data across load modules unless SAVE_DATA is created as an external action block with shared memory.

In the vast majority of cases, you should still use view mapping, but there might be some cases where the above technique will allow you to easily share a small amount of temporary data between a large number of action blocks without needing to include it as data passed on all USE statements.

64 bit conversion

2010-11-03T15:11:00.005Z

Gen r8 introduces the first platform to support 64 bit C code, which is HP Itanium. For the next release of our products, we will be using Gen r8 for Itanium and hence have had to port to 64 bit.

The UNIX source code generated by Gen is not specific to a particular UNIX implementation, so the same code is compiled for 32 bit on AIX and PA-RISC and 64 bit for Itanium. The difference is in the compiler options used.

One difference in the Gen r8 generated C code is that the variable used for the repeating group view 'last' flag has changed from an int to a long. In 32 bit architectures, an int and a long are both 32 bits, whereas for 64 bit, an int is still 32 bits but a long is 64 bits for the LP64 architecture used in UNIX (but still 32 bit for the LLP64 architecture used by Windows IA-64).

This means that EAB code must be modified to change an int to a long for the repeating group view variables in import and export views. You will also need to look through the EAB code to see if you have used int and long incorrectly since they are no longer the same. The same is true for pointers, which become 64 bits in both LP64 and LLP64 architectures.

Changing attribute lengths

2010-10-11T14:10:00.004+01:00

We recently decided to increase the length of a database column (attribute) to support longer path lengths. In principle this is a very easy task:

a) Change the attribute length
b) Amend the column length in the database design
c) Use database ALTER statements to change the physical database column length
d) Re-generate the affected code

In practice however, two aspects of the change were trickier:

1) Where the attribute is referenced in external action blocks, these will need to be identified and modified.
2) Code that is dependant on the attribute length might need to be modified.

The first issue was easy to solve. We created a custom function in Object List+ that lists all external action blocks that reference the attribute in an import or export view. The resulting list was then copied to a GuardIEn Change Request and then opened in XOS. All of the affected externals could then be downloaded and modified.

The second issue was harder. We had some code that assumed the old length of the attribute (in this case 50), for example, SET text = SUBSTR(attribute,49,2) was supposed to return the last two characters of the attribute. Now I agree that this is not great code, and the attribute length could be referenced using the length function rather than 50, but it was assumed that the length would not change and the hard-coded value used instead of the length to improve performance.

To identify these occurrences, a new VerifIEr check was developed that scans for code that uses the length of an attribute as a hard-coded value. This was able to identify code that needed to be changed and can also identify any future occurrences of this style of coding that would not be tolerant of a change in attribute length.

This illustrates one of the strengths of CA Gen. Because the action diagram 'source code' is stored in a SQL database using a precise structure (as opposed to the text files used by almost any other development tool), it supports complex queries that can scan the action diagrams looking for specific coding constructs.

Of Mice and Men

2010-09-23T14:35:00.006+01:00

Slightly off topic, but this might be of interest to older Gen developers. The Gen toolset is very 'mouse intensive' and much work needs to be done with the mouse rather than the keyboard. After 20 years of this, some of us older Gen developers are starting to feel the strain (literally), with RSI type irritations.

I found that changing to a different type of mouse was very helpful, and after trying a few out, now use a Vertical Mouse (see http://www.evoluent.com/). You may prefer a different style of mouse, and perhaps the main benefit is to change to something different?

Multi Row Fetch Experiences (3)

2010-09-01T16:41:00.003+01:00

In previous postings I have described how we have converted all of our READ EACH statements to use multi-row fetch. Also discussed were the results of a test that showed a significant performance improvement for a simple example which only had a single READ EACH statement. These improvements were extreme because a normal application will perform a lot more processing than simply the SQL for the READ statement.

On a real world example for a complex impact analysis, we have found an 18% reduction in elapsed time, which is a significant and worthwhile improvement given the low cost of implementing the changes to the model, especially since we have automated the setting of the multi-row fetch property using VerifIEr.

Parallel Generation Results

2010-09-01T11:49:00.004+01:00

We have now implemented the new CSE server. This has two 4-core processors and we recently conducted a simple test to benchmark the improvements gained when running CSE generation tasks in parallel (See previous post for introduction).

The results were that we obtained was a 60% reduction in elapsed time when running 4 threads in parallel and 70% reduction for 8 threads.

This was obtained with no other processes running, so for normal use, we plan to restrict a single generate task to a maximum of 4 parallel generation threads because of other tasks and on-line server processing requirements.

Parallel Processing

2010-07-30T12:11:00.002+01:00

As raw CPU speeds plateau, servers now achieve improved performance through multi-processor and multi-core architectures. This is ideal for servers that handle lots of separate processes like transaction processing or web requests, but what about large batch jobs? These tend to be designed as single threaded, so can only run as a single process for the application and a separate process for the database manager.

One of the recent enhancements we have been working on for the next release of GuardIEn is to reduce the overall elapsed time for large generation jobs on the client/server encyclopaedia by running multiple generates in parallel, thus taking advantage of multi-core servers. (We had already enabled parallel generates on the host encyclopaedia some years ago, which was implemented by submitting multiple jobs).

Because our tools are developed with CA Gen, we needed to work out the best way of implementing a parallel processing architecture.

There are several design alternatives to enable multi-thread processing for a CA Gen application. For this requirement, we decided to create child processes launched from the Gen action block and have the Gen AB wait for the child processes to complete. This enabled the design to launch a variable number of parallel generates (controlled by a parameter) and issue another generate when the previous one completed. The creation of the child processes is performed by a C external.

On our current test server that only has 2 processors, we have noticed a 30% reduction in elapsed time, and we are due to implement a new server with 2 4-core processors, which we hope will show even better reductions in elpased times for large generation tasks.

Multi Row Fetch Experiences (2)

2010-06-29T15:05:00.002+01:00

A second issue with multi-row fetch (see previous posts) affects application design.

With a normal READ EACH, each row is fetched one at a time, so if the row fetched has been affected by previous processing within the READ EACH, the fetched row's column values will be up to date.

However with a multi-row fetch, blocks of n rows are fetched into an array at the same time. If you then update row n+1 whilst processing row n, then when you come to process row n+1, the values in the entity action views will not be the latest values since they are current as of the time that they were fetched and hence not include any updated values.

This should be a rare occurrence, but worth bearing in mind when deciding if multi-row fetch is applicable.

Multi Row Fetch Experiences (1)

2010-06-29T14:50:00.005+01:00

We have now started the development of the next release of our products using gen r8.0. One of the new features of r8.0 that we are looking forward to using is multi-row fetch because of the potential for serious performance improvements (see previous posting).

We have developed a new check in VerifIEr to calculate what the optimum fetch size should be for a READ EACH statement and then use this information to automatically update the READ EACH statement.

However our initial testing has highlighted some issues with multi-row fetch.

The first affects DB2 and relates to errors or warnings issued during the fetch. If there are any warnings or errors issued, then DB2 issues an sqlcode +354 and you have to issue further GET DIAGNOSTICS statements to get the particular warnings. We have found several instances of warnings related to truncation of data. The warning is an sqlcode 0 with sqlstate 1004. This was caused by having an attribute defined in the model that was shorter than the database column due to differences between the same column in the Host Ency and Client/Server Ency.

Because Gen does not check the sqlstate (only the sqlcode), without a multi-row fetch, you will never see the warning, but with a multi-row fetch, because the generated code does not handle the +354, the application terminates with a runtime error. Unfortunately you cannot tell what the cause was without amending the generated code to add in the GET DIAGNOSTICS statements.

So far we have been working through the warnings and eliminating them, but we are also considering a post processor for the generated code to add in the diagnostics to make debugging easier, or to ignore the +354 sqlcode if there are only warnings.

The second issue is described in the next posting.

Encyclopaedia Performance

2010-05-28T12:44:00.003+01:00

A few customers have asked us for information on improving performance on both the Gen host ency and client/server ency. Rather than try and write out a definitive list of things to look for, I thought it would be useful (and less onerous) to provide occasional posts on this subject.

If you have any ideas or experiences that you would like to share, then please add comments to the postings, or if you prefer, e-mail them and they can be included in a new post.

The areas that will be considered in this section include encyclopaedia server performance, reducing contention, working practices and other factors that affect the overall performance of the encyclopaedia.

To start things off:

HE & CSE:
1) Model size rather than overall ency size has a big impact on performance. Smaller models perform much better than larger ones for the same function, i.e. downloading the same subset from a small model will be much faster than a large model. The speed is roughly proportional to the model size.

2) Encourage 'right sizing' subsets and working practices that are efficient, like only generating changed objects rather than all action blocks in a load module.

CSE:
1) Use the support client and check the Object Cache setting. The default is far too low. We have ours set to 500,000

2) Database i/o has a major impact on performance. Does your DBMS have enough memory for caching? Oracle likes a lot!

HE:
Have you implemented DB2 type 2 indices for the encyclopaedia? Originally (many years ago now) these were not available for DB2, so if you have an old encyclopaedia, you may not have converted the indices to type 2. This can have a big impact on contention. Type 1 indices are not supported from DB2 v8 onwards, so if you are on v8, then this will have been taken care of.

Gen r8 z/OS Libraries for Batch

2010-05-10T13:14:00.003+01:00

Gen r8.0 introduces support for operations libraries on z/OS (zLIBs). These are DLL load modules that contain one or more action blocks. The zLIB load module contains the DB2 language interface module and this is environment specific, for example DSNCLI is required for CICS, DFHLI000 for IMS and DSNELI for batch.

Therefore, as with standalone dynamic action blocks, a separate version of the zLIB is required for each operating environment, typically one for on-line and server environments and one for batch. This is also true for dynamically linked RI triggers since these are implemented as DLL load modules.

With 'normal' dynamic action blocks, separate on-line and batch versions are created by installing an on-line or batch load module that uses the dynamic action block. This requires the on-line and batch load modules to reside in different business systems and each business system must have a separate load library so that the on-line and batch versions of the dynamic action block are linked into different libraries.

In contrast, zLIBs and dynamic RI load modules are installed as separate load modules rather than as part of a normal load module. At present with Gen r8.0 you cannot specify whether you want to install the zLIB for on-line or batch and hence you cannot create a batch version of the zLIB.

We have developed a solution for this dilemma with GuardIEn 8.0, which has support for linking both on-line and batch versions of zLIBs and RI DLL load modules. CA plan to address this limitation with a service pack for Gen r8.0.

Beyond Compare

2010-05-04T11:21:00.002+01:00

As a tools developer for CA Gen, the fact that we also develop our tools with CA Gen has meant that we tend to 'build not buy'. In other words, when we find the need for some additional tools support as part of our own Gen development projects, we enhance our own tools. This approach then leads to extra functionality in the products that is almost always useful to our customers as well.

However there are certain 3rd party tools and utilities that we have purchased runtime licences for instead of building ourselves. Examples include the diagramming OCX that we have used for creating the Object Structure, Life-Cycle and Model Architecture diagrams in GuardIEn, the ftp/sftp utilities and the file compare tool.

For the file compare tool, we have used the freely distributable WinDiff tool with the option for customers to replace this with their own favourite product. However Windiff is a fairly basic tool, and some time ago we replaced this for internal use with Beyond Compare 3 (BC3).

We like BC3 so much, that for the 8.0 release of our products, we have purchased additional licences to be able to distribute BC3 to our customers as well.

Gen r8 and z/OS

2010-04-28T08:58:00.005+01:00

The beta test for Gen r8 has now ended and we are finishing off the changes made to our products to support r8. We will be launching this release 8.0 in early May to coincide with the general availability of Gen 8.0.

The most significant changes that affected us were on the z/OS platform. The introduction of z/OS Libraries (OPSLIBs for z/OS), dynamic RI triggers and changes to the way that applications are linked affected many aspects of GuardIEn, especially in the areas of impact analysis and code construction.

In previous releases of Gen, the link-edit control cards were created from the Gen skeletons using file tailoring and then a single INCLUDE was added for the dialog manager, with the remaining modules included using autocall.

With Gen r8, the format of the link-edit control cards has changed. Instead of using autocall to resolve called action blocks, each non-compatibility Gen module referenced in the load module has a specific INCLUDE APPLOAD or IMPORT statement.

This means that if you create the link-edit control cards outside of Gen, you will have to replicate this approach. A new business system library is available which Gen populates with the link-edit control cards (called binder control cards using the new IBM terminology), so these are available if required.

Another change is that dynamic action blocks that are packaged into a z/OS Lib are now called using a literal instead of a variable, for example, in Gen r7 and previously, a call to action block AAAA would be implemented as:

09 AAAA-ID PIC X(8) VALUE 'AAAA'.
...

CALL AAAA-ID

In Gen r8, if AAAA is included in a z/LIB, this is now

CALL 'AAAA'

If you installing code using multiple models, then the use of external action block and external system load libraries must be carefully considered to ensure that dynamic action blocks packaged into a z/LIB are not found via autocall since the binder will then statically link the object modules instead of resolving them using the IMPORT statement.

Multi Row Fetch

2010-03-31T11:51:00.005+01:00

One of the new features in Gen r8 that we have started to test is the ability to specify a multi-row fetch for READ EACH statements. This property changes the generated code to perform a block fetch of multiple rows into an array rather than fetching one row at a time.

A simple test of reading many rows from a large table using multi row fetch provided a 60% reduction in CPU compared with a standard READ EACH table. (The test was a DB2/COBOL batch job without much code apart from the READ EACH, so a real world example will show less of a gain)

The test compared various array sizes of 100, 1000 and 10000. There was very little difference in the cpu consumed between these, showing that there seems to be little benefit in setting a very large array size.

When we upgrade our own code to using Gen r8 for the next release, we plan to update all READ EACH statements to use this new option. To help do this, we have developed a new VerifIEr check that ensures that all READ EACH statements have this option specified and a corresponding genIE automated fix to set the property value. This intelligently inspects the code within the READ EACH to determine what array size to use, and it can also have an upper limit defined.

Build Tool Profile - Oracle

2010-02-11T10:33:00.002Z

We have noticed that the Gen Build Tool uses different Oracle pre-compiler options for UNIX vs Windows. Here are the default options:

Windows:
SQLCHECK=SYNTAX MODE=ANSI IRECLEN=255 ORECLEN=255 LTYPE=NONE

UNIX:
IRECLEN=511 ORECLEN=511 CODE=ANSI_C LTYPE=NONE MODE=ANSI DBMS=V7
SQLCHECK=SYNTAX PARSE=NONE RELEASE_CURSOR=NO HOLD_CURSOR=YES

We are not sure why RELEASE_CURSOR=NO and HOLD_CURSOR=YES are not set for Windows. We have experimented with setting these for Windows and have noticed a significant performance improvement.

Application Cache

2010-01-25T13:44:00.002Z

If you frequently need to read the same data from the database in the same transaction (for example look-up table data), the DBMS cache/buffer should reduce the I/O to disk by having the data in memory. However there is still a considerable overhead involved with using SQL to obtain the data multiple times.

A technique that we have used to improve performance is an application cache. This uses a common action block to read the data. The action block checks to see if the data is cached, and if it is, avoids the need for a READ from the database. If it is not available, it reads the data and then stores it in the cache.The cache is implemented using uninitialised local views and you can store as many rows as you are prepared to allocate memory to.

The design needs to ensure that the cache is initialised at the start of the transaction (for C generated code when using the TE) and also to cater for the possibility of updates to the database, i.e. the cache needs refreshing or deleting if the data can be changed during the transaction.

In specific examples where we have used the cache for heavily accessed tables, we have found the cache improves the performance of the READ action block by 1000%.

Mapping Group Views

2010-01-13T15:41:00.002Z

Gen allows you to view map group views with differing cardinalities on USE statements and dialog flows if the receiving view has a higher cardinality than the sending view.

However the view match remains intact even if you then subsequently change the cardinality of the sending view to a value that is greater than the receiving view, so you could end up with a sending view that is larger than the receiving view, which could then cause unexpected results, like loss of data without a runtime error. In this case, you could not establish the view match again, but the existing view match is still 'valid' in the model.

If the group view sizes were initially the same, the developer might not think that they need to add in any extra validation logic, but a subsequent change to one of the group views might then cause problems.

A new check in VerifIEr allows a quick check for differing group view cardinalities with a warning if they differ but are valid and an error if they differ and are invalid.

Discovering hidden errors

2009-12-21T15:45:00.002Z

Since we originally developed our VerifIEr QA and code checking product over 2 years ago, the scope of checks has enlarged in some very interesting directions.

The initial checks were primarily aimed at standards enforcement, for example, object names, CBD architecture enforcement, use (or abuse) of various properties (e.g. READ properties), etc.

More recently however we have developed a number of checks for customers that are aimed at improving code quality by detecting errors that might be otherwise difficult to find.

We have run these checks on our own products (which are developed with Gen) and have been surprised at the number of potential errors that have been encountered. Usually the more serious errors are detected during testing, but sometimes not!

Examples of the checks that we have found especially useful include:

Local views that are referenced but never set to a value, indicating that either the local views need to be populated or the code is no longer required;
Hidden fields (for example fields on a GUI that are placed above the top border) that are not read-only (the user could therefore tab to the field and change its value);
Export views not fully mapped to an import view on a screen or window;
Checking the tab sequence for GUI windows and dialog boxes;

These sorts of errors are notoriously difficult to spot via code inspection and can also be missed during testing.

Gen r8 Beta Testing

2009-12-09T15:00:00.002Z

October and November were busy months with 3 Edge conferences to attend and the start of the Gen r8 beta test period, so apologies for no blog posting for the past few weeks.

We have started our beta testing of the next Gen release (r8). There are a large number of new and changed features that will keep us busy with the development of the 8.0 release of our products that will support Gen r8.

A number of customers have asked about the availability of IET products to support their beta testing of Gen r8. Unfortunately we have only been able to start development of Release 8.0 of our products with the start of the beta programme and hence cannot offer a beta version of our products until this development work is complete.

Unused and Duplicate Prompts

2009-10-22T14:14:00.002+01:00

We have been developing our products for close to 20 years now and one of the consequences has been that we have found quite a few unused and duplicate prompts in the models. We also have a multiple model architecture and a policy of migrating the entire data model to each of the development models. This results in all of the prompts also being duplicated (and unused) in all of the models.

Apart from having a large number of redundant prompts in the models, it can also make the selection of prompts in the window/screen designer tedious because the large lists of unused and duplicate prompts makes locating the desired prompt harder.

There is a Gen function in the toolset to delete unused prompts, but this requires the model to be downloaded, and ours are too big. It will also not get rid of duplicates.

We have therefore written a new genIE function to both delete unused and consolidate duplicate prompts.

The result is faster downloads because you are not downloading extra prompts and also easier selection of existing prompts in the window/screen designer.

RI Trigger Impact Analysis

2009-10-01T13:32:00.001+01:00

The previous post discussed the need to regenerate programs that call changed RI triggers. The difficulty is in performing this impact analysis.

The Gen model does contain associations between action blocks and RI triggers (using IMPUSE associations if you are familiar with the Gen schema), but the important point to note is that these associations are maintained by the code generators. This means that if you change the data model, the IMPUSE association data in the model is not accurate until you have regenerated the affected code, which makes it useless in helping you understand what you need to regenerate!

We faced this issue when developing the impact analysis process that GuardIEn performs when it detects data model changes. To work out what RI triggers are directly affected by a data model change is straight forward, but the consequential impact on other triggers and action block / procedure steps involves a complex navigation though the data model, following cascade delete chains for example. It is also affected by the choice of generated or DBMS enforced RI rules.

RI Triggers

2009-09-29T10:15:00.000+01:00

In my experience, one of the more misunderstood aspects of Gen development is RI triggers, and the impact of making a change in the data model.

In many cases, users think that they only need to regenerate the RI triggers once they have made their data model changes. Unfortunately it is more complex than that.

The correct process is:

1) Implement any changes to table or column names in the data structure
2) Run the Ref. Integrity Process (accessed from the toolset Design menu) to synchronise the technical design RI constraints with the data model
3) Generate the affected RI triggers
4) Re-generate all action blocks that call the re-generated RI triggers
5) Install re-generated code

Steps (1) and (2) can either be performed separately or accomplished using the Retransformation tool.

The reason why you need to regenerate the action blocks that call the RI triggers is that the code generated for a DELETE, DISASSOCIATE, TRANSFER, etc. depends on the RI rules in the model. If these rules change (for example changing a relationship from cascade delete to pendent delete), then the action blocks require regeneration as well as the RI triggers.

Where is your source code?

2009-09-22T15:56:00.005+01:00

For your Gen applications, the answer is of course that it is stored in the Gen encyclopaedia. But what about your non-Gen code like:

External Action Blocks
OLE files
Bitmaps, icons
DDL
Documentation
etc.

Unfortunately we often come across projects that do not adequately manage the non-Gen source code, probably because they do not have this issue with the Gen code. There are cases of projects where they have lost their EAB source code and do not even have documentation for what the EAB did apart from the stubs in the Gen model.

Another common instance is where you use an OCX control in a GUI design. Gen creates a .ole file in the local workstation model sub-directory which contains properties for the OCX control. However this is not uploaded to the model, so if you delete the local model directory and have not saved the .ole files elsewhere, they are lost!

You should therefore take care that all project source code is properly managed. This could be as basic as ensuring that the files are stored centrally in a place where they will not be deleted, either accidentally or as part of a housekeeping routine.

Even better, the external code should be properly versioned controlled. There are many tools for this, ranging from simple and free source code control tools to more sophisticated products. Our own XOS tool has been designed specifically for managing Gen externals, including support for automatically versioning .ole files when a subset is uploaded to the encyclopaedia.

An unexpected feature of Object Migration

2009-08-26T09:17:00.003+01:00

An unexpected feature of the CA Gen object migration utility that sometimes catches us out is when you migrate an action block and the view matching of action blocks that use it is also affected.

Consider the example where AB1 uses AB2. If you add a new import view to AB2 and view match it to an existing view in AB1:

AB1:
USE AB2
IMPORTS: temp xxx to in xxx

and then migrate just AB2 to another model, if the view temp xxx in AB1 has common ancestry between the two models, then the view match is also migrated, which in effect modifies AB1 even though this had not been selected for migrate. However AB1 does not get a modified timestamp, so it looks like AB1 has not changed even though its view matching has.

This may not necessarily create a problem, but it does sometimes cause confusion.

Gen and null columns

2009-08-14T13:41:00.002+01:00

A recent posting on the Duick forum regarding NULL column support lead to a discussion on the Gen qualifier IS EQUIVALENT TO and a potential misinterpretation of the way that this works.

As a bid of background information, it is important to understand how a nullable column containing a NULL value behaves. Consider a table with a nullable column and these rows:

Id opt_column
1 ' ' (column has a value of spaces)
2 NULL(column is NULL
3 'X' (column has a value of X)

If you want a value of NULL to be equivalent to SPACES, and you want to read rows that have a space or null in the opt_column, then if your SQL was:

SELECT * FROM table WHERE opt_column = ' ';

would return just row 1 but

SELECT * FROM table WHERE opt_column = ' ' OR opt_column IS NULL;

would return rows 1 and 2

If you want to read rows that do not have ‘X’:

SELECT * FROM table WHERE opt_column != 'X';

would return just row 1 but

SELECT * FROM table WHERE opt_column != 'X' OR opt_column IS NULL;

would return rows 1 and 2

Once you understand the need for the IS NULL or IS NOT NULL qualifier in the SQL, you can write the READ qualifiers in the action diagram code.

The confusion arises over the use of the IS EQUIVALENT TO clause since it is likely that this does not work the way you expect!

For example, if an optional column has no value, then I think of SPACES & NULL as the same, so you would code:

READ table WHERE opt_column = SPACES OR opt_column IS NULL

However the statement READ table WHERE opt_column IS EQUIVALENT TO SPACES

gives the following SQL which is not the same:

SELECT opt_column FROM table
WHERE (opt_column = ' ' AND opt_column IS NOT NULL)

This means that if the column is NULL it will not return a row, which is the opposite of what I think you want.

IS EQUIVALENT TO is OK. For example, if you want a row with a specific value:

READ table WHERE opt_column = ‘X’ AND opt_column IS NOT NULL

Is the same as READ table WHERE opt_column IS EQUIVALENT TO ‘X’

which gives the following SQL:

SELECT opt_column FROM table
WHERE (opt_column = 'X' AND opt_column IS NOT NULL)

IS NOT EQUIVALENT also gives the desired result for a specific value but not for SPACES:

READ table WHERE opt_column NOT = ‘X’ OR opt_column IS NULL

and READ table WHERE opt_column IS NOT EQUIVALENT TO ‘X’ gives:

SELECT opt_column FROM table
WHERE (opt_column <> 'X' OR opt_column IS NULL)

But if you want a row where the column is not spaces, you would code:

READ table WHERE opt_column NOT = SPACES AND opt_column IS NOT NULL

but READ table WHERE opt_column IS NOT EQUIVALENT TO SPACES gives:

SELECT opt_column FROM table
WHERE (opt_column <> ' ' OR opt_column IS NULL)

In summary, it is best not to use EQUIVALENT with SPACES unless you want the behaviour that the generated code gives you. You would also need to be careful with a view that might have a value of spaces.

Documenting Changes

2009-08-13T10:30:00.000+01:00

Whether you are developing software that is commercially available or for internal use, users value a clear description of the enhancements and fixes introduced in a new release or service pack. This is especially important when there are changes in behaviour or actions that need to be taken by the users to take advantage of new features.

A customer recently complemented us on the quality of our release notes and asked whether we generated them from a database.?

Unfortunately there wasn’t a magic solution. We cut and paste the descriptions of each change into a Word document and then generate the PDF file from that. That part is simple though. The harder part is to ensure that each change is documented properly and is not accidentally omitted from the release notes. We therefore document the changes using the following process:

1) Each change must be documented in a form that will make sense to the end user, explaining the business reason for the enhancement or requirement for a fix. The documentation is in the long description of the Change Request (CR) in GuardIEn.

2) Each CR should address a single problem or enhancement. You should avoid CRs that span multiple requirements (the worst cases being a single CR that has all changes in it or a CR for changes made by a developer that is not linked to the actual requirements.

3) If a new requirement is found whilst changing some code, a new CR should be created for it and the temptation to ‘hide’ the new requirement within the scope of the existing CR avoided.

4) Once the CR has been completed and tested, the description should be reviewed for accuracy and any changes in behaviour noted.

5) The Release Notes should be updated with the CR and user documentation reviewed and updated as necessary. We have a separate state in the CR life-cycle to indicate that the documentation has been updated.

Almost like having a new machine

2009-07-14T16:17:00.006+01:00

We recently upgraded our anti-virus software to the latest 2010 release and it was immediately noticeable how much slower our machines were, especially our heavily used CSE machine. We also found the desktops to be much slower, and so the new release was de-installed and the older (2009) release re-installed. It still took up to 25% of the CPU though, and so we decided to try some alternatives. After a bit of research, we selected one of the other leading products to trial. Both desktops and the CSE run much faster and it is like having a new machine and so the upgrade can be delayed for a while!

In praise of integration

2009-06-29T12:34:00.003+01:00

Having spent over 20 years developing our products using Gen, it is clear that one of the main benefits is the low cost of maintaining applications developed with Gen. I think that there are many reasons for this, some of which are due to inherent features of Gen and others derive from the methods and standards used by the development project. In my view, a key feature of Gen that contributes to the low cost of maintenance is the integrated nature of the analysis and design tools.

The early marketing of IEF (as Gen was called in the early days) emphasised the integrated nature of the product and IEF was called an i-CASE (integrated Computer Aided Software Engineering) tool to distinguish it from point solution CASE tools. Unfortunately many i-CASE tools were nothing of the sort and few if any came close to delivering the 100% code generation and great success of Gen. This resulted in the CASE / i-CASE market getting a bad name, through little fault of IEF.

However. having chosen the best integrated development tool, shouldn’t a Gen project maximise the benefits of that integration? The trend to only use Gen for the server and batch parts of a project concerns me. Whilst there are undoubtedly situations where Gen is not the best choice for developing the user interface, I suspect that there are others where the choice not to use Gen for the front-end has been a mistake due to the resulting increased cost of development and maintenance.

When the user interface is developed with a separate tool, the interface between the presentation layer (client) and the business logic (server) has to become much more formalised at an early stage in the life-cycle, especially when the client and server parts are developed by separate teams. Even if you are using CBD/SOA or some other development approach that advocates stable, published interfaces, there are still many situations when a rapid, iterative approach to development will benefit from having one person develop the client and its closely coupled servers at the same time and with the same tool.

The goal of 100% code generation and integrated nature of Gen means that there are boundaries to the product's capabilities. Whilst there are features that allow external code (external action blocks, OCX controls, etc.), there are still limitations on what can be accomplished with Gen. The perceived weakness of Gen for developing sophisticated user interfaces has made some Gen projects avoid Gen for the user interface or presentation layer of an application.

A few years ago, I was visiting a long standing Gen user who had used Gen very successfully to develop 3270 and batch applications. I demonstrated GuardIEn to the development manager, and then we went for lunch. He explained that they were now moving to client/server but had decided not to use Gen for the front end because they did not think that you could develop a good front end. I asked him what they were looking for, and his response was that they would like to be able to develop something that looked like GuardIEn! He did not realise that GuardIEn was a Gen developed application with the user interface created using the same Gen design tools that they had decided were inappropriate.

Now, to achieve the sophisticated look and feel of our products with Gen has not been easy. We have had to develop an add-on tool (IETeGUI) and learn how best to achieve the desired results. But is this not the case with any tool? Don’t just take the product out of the box and expect to develop a very sophisticated user interface immediately. It needs quite a bit more work than that – probably more than you would expect. It is not easy to create a great user interface with Gen, but it can and has been done, and in my view, the extra effort is more than compensated for by the significant reduction in development and maintenance effort through the use of an integrated tool with 100% code generation.

Dog Food or Champagne?

2009-06-15T10:22:00.002+01:00

There is a saying about eating your own dog food, or the more pleasant version, drinking your own champagne. The point is that if you really believe in your own product, then you would use it yourself, and therefore I prefer the dog food analogy since you would only eat your own dog food if it was really palatable, whereas you might be prepared to drink anything that is alcoholic!

Anyway, getting back to the main point, if you are a software developer and you can use your own products, then you have a big incentive to improve them for your own benefit. This is why I was really pleased when I heard that CA would be using Gen within their development team as part of the Mainframe 2.0 initiative.

Because we develop our products with Gen, we are also able to use our own tools as well, and this positive feedback loop has resulted in many improvements and enhancements to make the ‘dog food’ as palatable as possible. An example of this is in the area of version control.

One of the most useful tools in the armoury of a developer is the ability to see what has changed in the source code. The ability to see the what, why, when and who (what has changed, why was it changed, when was the change made and who made it) makes diagnosing a problem much easier. With Gen, a single model can only contain a single version of an object, so if the object is changed, you lose the ability to see what it looked like the moment before the change, unless you have saved the previous version somehow (via migration, model copy, etc.).

Since it is impractical to save the previous version every time a change is made, often the diagnosis of a problem is made unnecessarily hard because this useful information is not available. For example, a user reports a problem in the test system that they noticed a few days ago. In the meantime, the model has been changed and you are therefore unable to see what the changes were (only that the object was last changed on a specific date/time). If you cannot reproduce the problem, you cannot then tell if the problem has been fixed, or if your test case does not properly test for the issue.

We have found the ‘minor versions’ feature of GuardIEn especially useful. This allows you to track every change made to a Gen object and see the who changed it, when it was changed and what was changed (down to properties and individual action diagram statements). When linked to a GuardIEn Change Request, you can also see why it was changed and what other objects were also affected by the same change.

I know that we would say this anyway, but we have found this capability to be invaluable in the on-going maintenance of our products.

Extending the GUI runtime

2009-05-27T13:12:00.006+01:00

Earlier postings have discussed the benefits of a runtime, for example, simplifying the development environment and allowing developers to concentrate on the business logic. We discussed how we have extended the runtime with custom action diagram functions. However the area where we have made the most use of a customised runtime is the GUI interface.

It is no secret that the standard Gen window designer offers limited features and the resulting look and feel often leaves users underwhelmed!

The majority of our products are client/server applications with a Windows GUI presentation layer designed in Gen. A few years ago we decided that it was important to improve the usability of the products and one of the key areas that we felt needing enhancement was the GUI front-end.

Some of the areas that we felt could be improved were:

Resizable windows and dialog boxes. These have to be designed for a minimum screen resolution (for example 800x600), but if a user has a larger monitor, it would be useful for them to be able to have larger window/dialog boxes. Two issues in a Gen application are a) if you resize a window, you then need a lot of code to reposition/resize the controls on the dialog box and b) the new size should be remembered since the user will not want to resize the dialog box each time they open it.

List Boxes. Most windows applications utilise list boxes where the user can change column widths, re-sequence columns, sort on columns by clicking on the column heading. Often a bitmap will make the data much more readable.

Tabs. These can reduce the complexity of an application that requries a lot of dialog boxes.

We investigated various approaches to implementing the above requirements. Most solutions involved a lot of additional code (for example to move controls around when a window is resized or to manipulate an OCX control) and would have involved a lot of effort. We also wanted to keep a common look and feel across all parts of the products and hence if we decided to enhance the products, we would have to re-code all of the windows and dialogs and not just the commonly used ones.

This is where we come back to the concept of a runtime. In our normal use of Gen, we did not have to add in a lot of window handling code – this was handled by the Gen GUI runtime. What if this runtime were extended to support our additional requirements? It transpired that a Gen list box could be ‘enhanced’ via Windows functions and still appear as a normal list box to the Gen GUI runtime. We were therefore able to extensively enhance and improve the functionality of the Gen GUI without needing to change either the action diagram code or the Gen GUI runtime.

Simple NOTE statements in the action diagram were used to provide parameters to the extended runtime and we were able to enhance the entire product line in a matter of days because the complexity was ‘hidden’ in the runtime functions and did not have to be coded by the developer.

The example below illustrates a window that utilises enhanced list boxes that can display bitmaps, it can be resized and the list boxes are resized automatically to fill the window, and various tabs are available with associated bitmaps as well.

The results of this effort are now available to other Gen sites as the IETeGUI product.

Beware hidden READ properties

2009-05-18T12:12:00.004+01:00

We encountered a problem recently where some code worked fine on our main Oracle test system but did not work correctly on z/OS DB2.

We could not see anything wrong, so ran the code in trace using xTrace and it was then that it was noticed that there was a LIMIT 1 clause set on a READ EACH properties. This limits the rows returned to 1, but only for MS/SQL or DB2. The statement should not have had the property set, but had been copied from another READ EACH that did have the LIMIT since there was an ESCAPE after the first READ and hence it worked correctly in all databases.

This highlights the danger of using properties that affect the results set, for example, LIMIT or DISTINCT since these are not visible to the designer when viewing the action diagram.

My recommendation would be that if you decide to use these properties, add a NOTE statement indicating this above the statement. This will act as a warning if the statement is copied, and also should also indicate that the properties might need modifying.

Client/Server Versioning

2009-05-13T11:43:00.001+01:00

Our products are designed as client/server applications where the client code resides on the user’s workstation and the server code resides on a central windows, UNIX or z/OS server.

The majority of our customers install the client code on each user’s workstation, though it is also possible to install it on a central file server. Because the changes to the workstation software are often strictly controlled and require the involvement of desktop support departments, in many sites there are restrictions on the timing and frequency that desktop software can be updated.

With a Gen client/server application, if the interface to a server changes (i.e. import/export views), then the client window manager needs to be re-generated and therefore a new version of the client software is required.

This presents two issues with the development and deployment of new releases of software.

The first issue is to prevent the client software invoking the wrong version of the server software. For example, if the server software is updated and a user attempts to access it with an older version of the client software, the software might fail (typically with a common format buffer error) or even worse, complete execution but with incorrect results.

The solution that we have adopted to the first issue is to incorporate a version number in the client software . This is passed to the server on each server call and the first thing that each server does is to verify that the client version number is compatible with the server. The version number is nearly always the same as the major release of the software and hence this ensures that the client and server software is at the same release, for example 7.7.

The second issue is to allow fixes and minor updates to the software within a release that does not require a complete refresh of the client software on all user’s workstations.

To address this requirement, we do not make any changes to the server interface (import/export views, view properties, transaction codes, etc) within a major release that would require a new client version. This means that the server code (and client code if required) can be updated to the latest service pack or with emergency fixes without needing to re-distribute or update the client code.

If there is a real need for a new client/server flow within a release, this can be accommodated with a completely new server procedure step. Updated clients can access the new p-step, but existing clients will be unaware of it and hence continue to operate unaffected. With this approach, the only consideration is that the server code will need to be deployed before the client code.

Single Logon

2009-05-01T12:04:00.002+01:00

When launching our products (GUI client/server or local windows based ‘fat clients’) the user has to logon by providing a userid and password. These are then used to validate that the user has access to the system and optionally are also provided to the Gen client/server runtime modules for authentication at the server level, for example, to RACF on the mainframe.

Since we have multiple applications available via a program folder or accessed via the Gen toolset plug-in menu, we wanted to avoid having to ask the user to logon each time they started a new application if another one was already open.

This meant that we needed to find a way of sharing the logon credentials. This method had to be secure and reliable, so storing the values in an environment variable or file was ruled out. The solution we adopted was to use a DLL that was shared by all of the clients and the user credentials stored in the DLL’s memory with encryption.

Normally if multiple executables share the same DLL, each instance will have its own private copy of the memory, thus precluding the sharing that we were trying to achieve. The solution was to use a shared memory segment in the DLL so that the contents of the shared segment could be accessed by all executables.

The result is that once the user has logged on to one client, then all subsequent clients do not ask for the userid and password. An additional benefit of the shared DLL is that when the last client is closed, the DLL is no longer resident in memory unlike using persistent storage methods.

Memory Usage Update

2009-04-24T12:18:00.001+01:00

A previous post described some of the techniques that we have used to reduce the memory requirements of Gen applications. One of the approaches that we have been trialling is to delay the allocation of local view storage for generated C code.

The generated C code for an action block will define a structure for the local views and the memory for these structures will be allocated for all of the action blocks linked into the load module when the load module is loaded, even if the action blocks are not called. Note that this is only strictly true for action blocks that do not have any uninitialized local views.

We have developed a source code post-processor that re-defines the local view structure so that the memory is only allocated when the action block is called and is then freed on return from the action block.

This has dramatically reduced the memory usage of the load modules, especially for co-operative servers that remain loaded by the transaction enabler.

We have been running with the modified code for a month now with no unexpected results. The benefits are a big reduction in the overall memory requirement. This allows us to improve performance by a combination of having more load modules loaded into the TE thus reducing program load time, freeing up memory that can then be allocated to the DBMS and fewer page faults.

User Defined Functions

2009-04-16T13:05:00.003+01:00

In my last post, I described how you can extend the Gen runtime with user defined functions.

The post described how the use of user defined functions can simplify action diagrams because you can nest functions and also the input and output parameters can be specified using any attribute view of the same domain, whereas imports and exports to action blocks have to use the same entity type or workset views.

It is worth pointing out that whilst this simplifies the action diagram code, the underlying generated code will still need to ‘map’ the input and output views to additional data items that have the same length as the function’s parameters, and hence the fact that the action diagram code appears not to have this extra ‘mapping’ does not mean that there isn’t some runtime overhead associated to this data mapping.

To add a function, you have to code the logic in the same way that you would for an external action block and place the code into a library so that it is available at link time. You also have to define the function to the model, specifying the input and output parameters. These are more restrictive than for a normal action block. You are limited to one output parameter and up to 5 input parameters, and group views are not allowed. Defining the function to the model is quite a complex process, especially if you are not familiar with the Gen meta-model, so you may want to take a look at the QAT Function Manager which automates much of the tedious tasks associated with defining the function. There are also various presentations and documents produced by CA on this subject, so I do not want to repeat them here.

Once you have added your function to the model and coded the logic in C, COBOL or your other target language, you are ready to start using it in the model.

Some of the functions that we have added are:

IET_FIND. This is a replacement for the standard Gen find function which allows you to search a string using wild cards (like %) and other search operators.

IET_GET_ROW_CLICKED. This is a function that returns the clicked row given an x & y co-ordinate returned by the Gen GUI right mouse functions and a list box control. Gen only provides the x & y co-ordinates and to get the row selected, you need to adjust for the scrolling of the list box, the font height and whether the user has changed the default DPI setting in Windows.

IETeGUI functions. For IETeGUI we have developed a range of functions that allow the developer to fine-tune the GUI at execution time, for example, dynamically changing a list box column position, visibility, headings, etc. Using functions greatly simplies the coding when you want to provide detailed control over the GUI runtime.

Extending the Gen Runtime

2009-04-14T16:09:00.002+01:00

I thought it would be worthwhile describing how we have extended the Gen runtime in the development of our tools, but first we should clarify what we mean by the ‘runtime’.

A high level language (including Gen, but also languages like C, COBOL, Java, .Net, etc,) simplifies the developer’s task by providing functions that are inherent in the programming language. Examples include printf in C, STRING in COBOL and the Gen concat function.

The actual implementation of the function by the compiler (or code generator) might be via a macro, code generation (i.e. the COBOL or C compiler generating assembler instructions to implement the statement) or via a call to a language runtime module. To the developer, it does not matter how the function is implemented. For example, the CA Gen concat function is implemented in generated COBOL code whereas in C it is implemented via a Gen runtime function call.

So what is the difference between a language’s runtime functions and those that you might procure from a 3rd party vendor or develop in-house?

From a developer’s perspective, in some languages (like C) the code to use a language function will not differ much from the code to use a 3rd party function, whereas in others (like COBOL), the in-built functions are typically referenced via language statements (i.e. STRING) whereas other functions would be used via a CALL statement. Typically a function that is invoked via simple language statements is easier to use than having to implement a separate program call to invoke the function.

In CA Gen action diagramming, the same is true. Compare the following:

SET c to numtext(substr(concat(a,b),1,5))

With the following which uses alternative routines for numtext, substr and concat.

SET map_1 TO a
SET map_2 TO b
USE my_concat
WHICH IMPORTS: map_1, map_2
WHICH EXPORTS: map_3
SET map_4 TO map_3
SET start to 1
SET end to 5
USE my_substr
WHICH IMPORTS map_4,start,end
WHICH EXPORTS map_5
SET map_6 to map_5
USE my_numtext
WHICH IMPORTS map_6
WHICH EXPORTS map_7
SET c to map_7

In the first example, one statement could achieve the desired result, whereas if we replace the use of in-built functions with external action blocks that achieve the same result, we need a lot of additional code as well as special ‘mapper’ views that are of the same type as the import and export views of the EABs.

In the above example, I have shown an extreme case where each function has its own set of import and export vies that are not shared between the functions.

Gen allows you to develop your own functions, so you could then code the above as:

SET c to my_numtext(my_substr(my_concat(a,b),1,5))

We have made use of this capability to add our own functions into the Gen model and hence simplify our code, both in terms of the number of statements needed to implement a given task, but also reducing the number of ‘mapping’ views needed.

In a subsequent posting, I will describe some of the functions that we have developed.

Windows Timers in Gen GUI

2009-04-09T13:24:00.002+01:00

Watching paint dry is not everyone’s idea of an interesting day at work, and for certain applications this is extra tedius when you have to click a ‘refresh’ button just to see how dry that paint has become.

Various functions in our applications submit background tasks and exhibit this mindless click-to-see-if-it-has-finished scenario. After much RSI and boredom we decided it was time to implement auto-refresh and timer events in the GUI so that at least you didn’t have to click the tired old mouse so much.

To achieve this in our Gen GUI we developed an EAB which we call to register and start a timer event in Windows (WM_TIMER messages). Basically this is a Windows event interception routine which processes the timer events we are interested in, and any other events get passed on to the default window event handler (the Gen GUI & Windows runtimes).

To make something actually ‘happen’ when the WM_TIMER event is received, TIREVENT is used to trigger an event in the client PStep, and the timer event is deactivated again, as far as Windows is concerned.

The event in the PStep can then do its' thing, and if it wants to repeat the timer again (like a regular auto-refresh) it simply calls the EAB again telling it to reactivate the timer.

Pretty simple stuff in principle, but a big improvement for the poor user and their aching click-finger.

Tabs Controls in Gen GUI – an easier approach

2009-04-09T13:18:00.004+01:00

Let’s face it, designing neat but sophisticated applications using Gen’s GUI can be challenging. The temptation to resort to OCX controls to get a good-looking app is high, but at the expense of difficult and cumbersome coding in your client PStep.

In particular, a user-friendly way to simplify groups of activities or information in a GUI would be to use a Tab control. If you were coding any other language (VB, C++ etc.) this would be an easy choice, but in Gen the result is a huge client PStep which contains all the logic for every tab page.

Better would be to split each tab page’s logic into a separate client PStep, but this simply is not possible using OCX controls in Gen.

To get the look and feel of a tab control, we’ve developed some cheats which allow exactly this… very simply, each tab IS a separate client PStep, but the tabs are actually a combination of buttons and bitmaps, designed to look like tabs!

To reduce the complexity of the Dialog Flows required, a central non-screened PStep is created which has the Dialog Flows (links) from itself to each of the tab PSteps. There are no flows between the tabs themselves. If a tab button is pressed, it simply flows back to the non-screened PStep with instructions on where to flow to next.

There are of course a few tricks to make this look slicker, including positioning and sizing of the individual tab windows, to give the impression that they are in fact the same window. Originally we achieved this by simply getting the window x,y,w,h and passing it around on the flows, but now we take advantage of IETeGUI’s resize features specifically for this purpose.

Find out more about IETeGUI here... http://www.iet.co.uk/Products/IETeGUI

Also, the tab buttons tend to ‘float’ around a bit, compared to the designed position in the Gen Toolset meaning that the buttons wouldn't always be perfectly lined up, so we put logic into each client PStep to explicitly place them at specific positions so they line up nicely. Again, we now make use of some automatic functions in IETeGUI to help with this.

Another aspect of the illusion of tabs is the border of the tab control. Our original implementation simply used carefully designed background bitmaps for those windows, and the tab buttons were placed so that they sat along the top border of the tab. If your windows are not resizeable, then this works perfectly. However, for resizeable windows, you’ll want to resize the tab’s border as the window gets resized. To achieve this we originally wrote some C code which intercepted the window’s redraw/resize events and drew the border dynamically, but now use some built-in features of IETeGUI to achieve much the same thing.

Avoiding too much logic in Window Open events

2009-04-09T13:11:00.003+01:00

When flowing between windows in a Gen GUI application, it is good to have the window pop-up quickly, even if the data is not populated immediately, as this gives the user the feeling that something is happening.

If you put too much logic inside the Open event of the window, this perception changes because the window does not get physically displayed until the end of the Open event, thus giving the illusion that the application has hung, crashed or simply disappeared.

We’ve found that a good practise is to minimise the amount of logic (especially server calls) performed in the Open event, and if necessary use a TIREVENT within the Open event to trigger another event where the longer processing is performed. The window will then get displayed after the Open event so the user can at least see it is alive.

Coping with the 32k view size limit

2009-04-06T11:03:00.003+01:00

You are likely to have come across the 32k view size limit for procedure step import and export views unless you are generating for platforms that do not have the limit like pure windows clients, Java or .Net.

When the user needs to scroll through simple lists of data, a paging method that uses a ‘start from’ field is usually adequate, and if you want to display a lot of data on the client, you can still repeatedly call the server to retrieve batches of data, again using a hidden ‘start from’ field.

However there are occasionally situations where the above approach does not work. Examples that we have come across include impact analysis on the server which returns a set of object ids based on a complex trawl through the database. You need to return the complete set of data and the 32k view size limit can then result in truncation of the results.

We have therefore developed some techniques for handling this.

The first technique is to limit the export view to the identifiers of the returned rows. You can then call a different server to retrieve additional details, for example, the first server returns 1000 identifiers and then a second server is used to retrieve the additional details in batches of 100.

The second technique is to reduce the number of attribute views in the export view. For each attribute in a view, Gen adds 1 byte for the state flag and for generated C code there is an additional null terminator byte. Gen also adds an additional 6 bytes for each attribute in the export view. Consider the situation where you need to return ten 1 byte attributes in a group view. For C code, each row in the group view will occupy 90 bytes. However if you were to string the ten bytes together into a single 10 byte field, then each row would occupy 18 bytes, thus allowing a much larger group view size.

The third technique is to first call the server to populate the result set and return the first batch of data. Any remaining data is stored in a temporary location, for example a file. If the results set is incomplete, the client can then call the server again repeatedly to retrieve the remainder of the data from the temporary location.

Memory Usage and OS differences

2009-03-27T11:26:00.002Z

The work that we have done so far on reducing memory usage has concentrated on the windows platform, but we also run our products on UNIX (AIX & HP) and z/OS.

Each platform is different with respect to when memory is allocated, so an understanding of this is important when trying to reduce memory usage, especially if your code will operate on different platforms.

The examples that we are using relate to generated C or COBOL code and not other platforms like Java or .Net.

For C code, the storage for local views is allocated on initial load of the executable. For static modules this will be when the executable is loaded, and for DLLs, this will also be when the executable is invoked. However, once it has been invoked, the memory will remain allocated until the executable terminates. For an executable loaded by the Transaction Enabler (TE), this might be some time!

You can avoid loading a DLL that might not be invoked for each execution with the /DELAYLOAD linker option on Windows and equivalent options for UNIX (e.g. -blazy on AIX). We are experimenting with this and will report back on findings in a later posting, but so far we have seen a benefit of Windows without a noticeable overhead. The main benefit on Windows is that we can now keep many more executables loaded in the TE and thus do not have the overhead of these being swapped in and out.

On z/OS, the COBOL working storage is allocated from the heap for re-entrant code. (Normally Gen programs are compiled with the RENT option). This means that the working storage is allocated on first execution of the action block and then not freed until the executable terminates. If you have an executable with a lot of action blocks that are rarely invoked, then they will not present a memory overhead compared with C code where the storage for all action blocks will be allocated on initial load of the executable.

Memory Usage Caveat

2009-03-25T13:05:00.003Z

In the Mythical Man Month, Fred Brooks provides an example of how a developer 'wasted' a few bytes of storage in the OS/360 control program by putting in a routine to handle leap years. In his view this was not necessary since the operator could reset the date once every 4 years. In those days memory was very expensive, and one of the main ways of charging for the machine, so freeing up a few bytes to the application program was seen as a good thing.

We are now used to machines with many gigabytes of memory and so you may be wondering why this blog started with a post on reducing memory usage, especially since memory is cheap and developers are expensive.

Let me start by stating that I am not advocating spending a lot of precious time trying to save a few bytes. In our case, because of large group views, large number of called action blocks and reuse of code in shared libraries, some of our server load modules were requiring over 100Mb. If you then have 20 of these loaded by the TE, you are now using 2Gb.

Paging memory to disk or swapping load modules in and out of the TE has a performance impact, so our thinking is that if you can reduce the memory usage of the application, then you can keep more load modules loaded and/or give more memory to the DBMS, which will be able to benefit from it and hence improve performance.

A second point is that most of the techniques that we are using are highly automated. We have developed tools to search for and remove unused views, and report on imperfect view matching (optionally only for repeated calls or those involving group views). There is therefore a very low overhead in implementing these good practices.

You should however consider the cost/benefit case for any coding standards that you adopt, and spending a lot of time trying to save a few bytes (or even a few Mb) is unlikely to be justifiable.

Memory Usage

2009-03-20T10:13:00.005Z

Reducing the memory usage of our load modules is something that we are working on at present, so to kick off the blog with a very technical subject...

Much of the code that we develop relies on repeating group views to process lists of objects.

Group views in Gen need to have a fixed, maximum size, and memory for the group view is allocated at program load time. This means that there is a trade-off between the memory usage of a load module and the limits imposed by the group view size. In many cases you can size the group view to cater for the maximum number of rows that are needed, or repeat processing with some sort of 'start from' value.

However there are quite a few instances where this is not possible, and then the group view size becomes a fixed limit on the data that can be processed by the function.

Over the years we have increased these limits based on customer requirements, and this has resulted in an increased memory requirement for the load modules. This is particularly noticeable in Windows and UNIX environments where the server load modules are kept in memory by the Transaction Enabler. (Note that there are tuning parameters for the number of load modules that are loaded by the TE).

We have also made more use of DLLs in Windows and UNIX (Operations Libraries) and these can also increase the memory requirement if they result in a load module ending up loading a DLL for one action block and therefore also allocating the memory for all of the other action blocks in the opslib and any dependent opslibs. Our models have a lot of shared and reused code, and complex dependencies between modules has meant that many of the DLLs are loaded multiple times because of code dependencies, even if they are not required for each execution.

We are therefore working on some strategies to reduce memory usage. Ones that we have come up with so far include:

1) Delete unused views and attributes
2) Improve use of perfect view matching
3) Reduce use of uninitialised local views
4) Share group views between action blocks
5) Delay loading of DLLs
6) Delay allocation of local view storage

The first four are all standard coding techniques.

1) Maintaining an action block over many years (some of our action blocks are now 20 years old) can result in quite a few unused views. Finding and deleting these can be tedious and time-consuming, so we developed some tools in VerifIEr and genIE to automate finding and then deleting the unused views and also unused attribute views.

2) If view matching is not perfect, extra working storage is required in the generated code, so improving view matching (especially for large group views) will help.

3) In C generated code, having even one uninitialised local view will double the memory usage of all of the local views, so avoid this if you can.

4) An example of sharing memory is where many action blocks require a large local group view with the same structure. For example, we have a case where many action blocks need a 1Mb group view and each is called from the same parent. If each action block has its own local view, then each AB needs 1Mb of storage. However if the group view is defined as an exportable import view and is passed from a local view in the parent, then they all share the same storage. You then need to ensure that the view is properly initialised, but there is a big memory saving. In one load module we managed to halve the memory usage using this technique.

5) On windows we are experimenting with using the /DELAYLOAD: linker option to delay loading of opslibs. If the route through the code never requires a function in the opslib, it will not get loaded.

6) This is very much work in progress, so more on this later...

Welcome

2009-03-20T09:06:00.003Z

Welcome to the IET Gen Development Blog. I decided to start this blog to share some of the tips and techniques that we have learnt at IET in developing our products with CA Gen. We are passionate about CA Gen and the main aim of this blog is to help other Gen users learn from our experiences (and mistakes!).

The blog is likely to be composed of random information and is not designed to be a structured training course in Gen development!