View Starving or Perfect View Matching?

A recent question from a customer was related to view starving and perfect view matching.

The technique of view starving involves removing all redundant attributes from views to avoid unnecessary storage allocation and view initialisation logic.

The technique of perfect view matching involves ensuring that the view structures in a USE statement are identical so that the program call can be achieved via passing of the actual views.

The question related to the problem that if you starve the views, you might not get perfect view matching, and hence should you then allocate a new local view that has perfect view matching and add in extra statements to manually maintain this temporary view to achieve perfect view matching for the USE statement.

If you do not have perfect view matching, then Gen needs to generate an intermediate data structure in the calling program to accomplish the parameter passing on the call of the used action block.


Data needs to then be passed from the actual views in the calling AB to the intermediate data structure and therefore there is an overhead for a) allocating the storage for the intermediate structures and b) the instructions to move the data.

Therefore in this situation, there is little difference between doing this yourself via a local view or getting Gen to automatically generate the extra structures – it amounts to the same thing, at least for one USE statement. Note that the order of the views is also important. Your local view would have to have the same attributes in the same order, and to work this out requires a careful comparison of the two views since the view matching dialog will not indicate if they are ordered differently.

It only becomes more efficient to define your own views if you have many USE statements that would make use of your fully populated view, since Gen would generate the data moves for each USE whereas you may only need to move the data once for multiple calls. However this becomes more complicated to understand and maintain. It only needs one attribute to be added, deleted or even moved in sequence for the technique to not accomplish perfect view matching, and then you get a double overhead – your views and the extra generated code. The additional code also complicates the action diagram, often for little benefit.

In an on-line transaction with a single USE statement, the overhead is not worth worrying about. You should concentrate on achieving perfect view matching where it will affect performance. This would typically be for large group views and repeated calls to the same action block, for example, in batch jobs where the same action block is called within a loop that is executed many times.

With our automated code checking tool VerifIEr, we have a perfect view matching check. This will work out whether you have perfect view matching or not, and can be configured to only check group views and USEs within loops, so that you only have to focus on the important ones. VerifIEr can also check that attributes are used, and hence help with view starving.

Where you want to have perfect view matching for performance reasons, I would add in the extra attributes to the ‘proper’ views rather than add in extra views, since there would be no benefit from adding in the extra views. The only time this might be needed is if there are many called ABs with different import views and all sourced from the same view in the caller. However in this situation I would then let Gen handle this situation rather than introduce my own additional views.

Trace Lite

When you can't work out what is going wrong with your code, you can often solve the issue by stepping through the statements using the Diagram Trace Utility (or xTrace on the z/OS platform). However this requires regeneration of the code with the trace option enabled and this takes a bit of time.

In many cases, just seeing which statements were (or weren't) executed is enough to locate the cause of the problem. You can now do this if you are using pathvIEw, since it has a feature to display the last executed statements.

Moving beyond the 32k CFB limit

Gen r8 IE2 increases the Common Format Buffer (CFB) limit from 32k to 16M.

The increase in the CFB limit is available for C generated applications on Windows and UNIX, but not in this release for z/OS, and hence we will not be able to take advantage of this for our products until z/OS support is available for CICS and IMS COBOL servers. It must be the top ranked enhancement request for CA Gen for the past 20+ years, so it is great to see this finally make it into the product.

Previous posts have discussed strategies for coping with the 32k view size limit, and our experiences with a Web View interface has been that it is a good idea for web applications to fetch and display small pages of data rather than trying to bring back a huge results set into a group view. The 32k limit therefore can be a good thing because it provides a limit on the amount of data returned in a single server call.

With the new limit, a developer could massively increase the data returned by a server to ~16M, which might be a good thing if the application did this anyway with repeated server calls, but a bad thing if the user was previously responsible for paging through the data and using filter/selection fields to limit the number of rows displayed, but they can now display thousands of rows without needing to worry about the filters.

Thinking about the impact on our own products (which are developed with Gen), there are several servers that would be much simpler with a larger export view size, so once z/OS support is available, we will take advantage of the larger view sizes to simplify the code and improve performance.

Gen 8 Interim Enhancement 2

We have just started beta testing Gen r8 Interim Enhancement 2 (IE2).

(Interim Enhancement is the new term within CA for Feature Pack).

The new features that we are particularly interested in are:

• Increase in the common format buffer (CFB) limit from 32k to 16M
• 64-bit Windows applications
• Customised Java proxies

Visualising Data

With the development of pathvIEw, we wanted to provide a graphical display of how complete the testing was for each module. To do this, we developed a new function in IETeGUI to provide a percentage complete indicator to display in a listbox cell, like this:

This got us thinking about other ways of visualising data to make it easy to glance at a table of data and see the most important bits. We therefore created two additional bar chart styles which are illustrated below. The first style is just a single bar and is used to illustrate the number of statements - the longer the bar, the more statements there are.

The second style is similar to the first, except that the colour is based on a medium and high threshold. We have used this for the complexity column, with green showing low complexity values, amber for medium complexity and red for high complexity.

The nice part about using IETeGUI is that the above is handled without the need for any additional OCX controls and is displayed in a standard Gen listbox.

Cyclomatic Complexity and CA Gen

As part of the development of pathvIEw, we have introduced a new complexity  metric which is calculated for each action block and procedure step.

We conducted some research into the usefulness of the metric and this shows some interesting results into the use of complexity counts for predicting error rates in code and hence for its use in prioritising testing efforts.

See the research document here: Cyclomatic Complexity and CA Gen.

Detecting Redundant Code

We recently launched pathvIEw - a new Code Coverage testing tool for Gen, and since using a product yourself is the best way of gathering ideas for enhancements and improvements (as well as testing it of course), we have been using pathvIEw as part of our testing to ensure that we have covered all of the paths through the logic.

When you find that some branch of the code has not been executed during the testing, this could be because your testing has not been thorough, but it could also mean that you cannot get to these statements, i.e. it is code that can never be executed and hence can be removed.

Whilst testing some new functions in pathvIEw recently, there was a block of code that had never been executed, and when I looked at the path through the logic to this code, it became apparent that it was never going to be executed, and hence could be removed.

Web View Experiences - View Sizes

Our GUI client design for list boxes is to retrieve all of the rows that satisfy the filter criteria and then allow the user to easily scroll up and down through the list box. Because of the 32k view size limit on the server procedure step, the client list box size is much larger than the server view size. For example, the client list box might allow up to 32000 rows (the limit for a GUI list box is 32767 rows) but the server group view size might be 100 rows, so the client would call the server multiple times to retrieve all of the data.

Whilst this approach works well for a GUI client, it presents performance problems for the web interface. Therefore our design for the web interface is to only call the server once to retrieve the first set of rows (for example 100) and then provide Prev and Next buttons on the web page to support paging.

The servers will already support the Next page operation because this would use a 'start from' field that would be set in the GUI client to page forward through the data. However our servers typically did not need to support a 'previous page' operation and so this has had to be added to the servers to support the web interface.

Web View Experiences - Introduction

In the latest release of GuardIEn (8.1) that we launched this month, we introduced a web based user interface for viewing and managing Change Requests and Release Packs. Called WebCR, the primary purpose for developing this new interface was to provide an easy way of giving a wider group of users access to the change management functions of GuardIEn without needing to install the full client software on their workstation.

All of our products are developed with CA Gen, and we also wanted to develop the web interface using Gen so that we could maintain all of our code in a single tool, and share common logic between the GUI client and web interfaces.

In the past we had experimented with Gen's Web Generation capabilities, but decided that these did not meet our requirements. However with the introduction of Web View in Gen 8.0, we decided to re-evaluate Gen's capabilities and started an R&D project with Web View. The results were promising and the decision was taken to proceed with the full scale development of a new web based user interface for the change management aspects of GuardIEn.

The design criteria was to provide similar functionality as the GUI client and use the same servers as the GUI, so that a customer wanting to use WebCR simply needs to install the WebCR application in an application server and the user interface is up and running.

Whilst we wanted to provide similar functionality to the GUI interface, our research project indicated that there were some important differences between GUI clients and Web clients with respect to dialog design, window layout and performance. This meant that we needed to have separate client procedure steps for the web interface, although they could share the same servers as the GUI client and also common code.

In subsequent posts, I will describe some of our experiences of using Web View in more detail.

RI Triggers – Gen or DBMS?

Sorry for the delay since the last posting, but we have been very busy at IET with launching a new release along with a couple of new products.

We recently had a discussion with a customer regarding the difference between Gen and DBMS RI enforcement.

The advantage of using DBMS RI is that the RI integrity is maintained by the database and not Gen generated code and hence any program or interactive SQL that deletes records will ensure RI integrity is maintained, whereas with Gen RI, you must either always perform deletes using Gen programs or ensure that your non-Gen programs or SQL correctly maintain RI integrity by cascade deleting child rows, setting foreign keys to NULL, etc.

However one important consideration is that many DBMS products do not support the full range of delete rules that can be defined in Gen. One example is a pendant delete, where the parent row is deleted when the last child is deleted. In this situation, Gen will enforce the rules that cannot be enforced by the DBMS, so that the generated RI triggers contain a mixture of Gen and DBMS enforced rules.

The danger with this situation is that you might think that all RI is enforced by the DBMS and hence not worry about deletes performed outside of Gen, however the DBMS would only be performing some of the deletes and hence the results would differ between using Gen to perform a delete compared with non-Gen programs.

Another consideration with DBMS RI is that you must ensure that the DBMS rules are kept up to date, on all databases, for example, development, test and production.

For these reasons, we use Gen enforced RI for our products.

GuardIEn and Versions

I was recently asked to clarify how GuardIEn handles code versions, and thought that posting the reply might be helpful to others.

GuardIEn stores meta-data about versions (i.e. information about the versions) rather than the versions themselves, and the actual versions of objects have to be stored as objects in Gen models. Hence to be able to ‘use’ any prior version in a Gen process like a migrate, it must exist in a model.

If you consider an example of a 3-level development hierarchy with DEV, TEST and PROD environments, then as long as you have a Gen model associated to each environment, you could have up to 3 separate versions of an object, one per model.

• If you start with the same copy of the object in each and assign it Version 1, then in GuardIEn there will be one version (V1) and that version is in the DEV, TEST & PROD models.
  
• You change the object in DEV and upload it using the Upload Assistant, GuardIEn will create a new version (V2) and assign it to the first state in the life-cycle for the DEV environment. You then have two versions in GuardIEn and two versions in the Gen encyclopaedia – V1 in TEST & PROD and V2 in DEV.
  
• You migrate the object from DEV to TEST and the status of V2 will be updated to a TEST state and you still have two versions, V1 in PROD and V2 in DEV & TEST.
  

If you want to backout the change in TEST, you could migrate the object from PROD to TEST (assuming that there were no other changes in the model that would prevent this migration) an reset the status of the version to a DEV state.

Once V2 is migrated to the PROD model, you no longer have a copy of V1 in any model and could not go back to V1 through migration.

It is often useful to be able to see what the contents of previous versions was even if you do not have them in a model, and to support this, we have support for Minor Versions. Whenever an object is changed via Upload Assistant or genIE, a copy of the object is taken as a text file (PAD listing for AB/PROC, objects & properties report for ENT/WAS, source code listing for XOs) and stored in the GuardIEn database. This provides a full audit trail of each change made to an object. Whereas in a Gen model you only see the latest copy of an object and hence the detailed who/what/when of multiple changes to the same object are lost in Gen, with Minor Versions you would see all of the changes separately.

This is then a very useful resource, not only as an audit trail of changes, but also for problem solving. The ability to see the what, why, when and who (what has changed, why was it changed, when was the change made and who made it) makes diagnosing a problem much easier. With Gen, a single model can only contain a single version of an object, so if the object is changed, you lose the ability to see what it looked like the moment before the change, unless you have saved the previous version somehow (via migration, model copy, etc.).

It is possible to configure GuardIEn to manage a backup model which is maintained as part of a system update to production. The section on Backup Migration in the System Updating Steps manual describes the process that the backup migrate step uses to scope the objects that are migrated to the backup model. However because a Gen model enforces strict consistency between objects, it may not be possible for the n-1 version of two objects to co-exist in the same model, for example, if the n-1 action block uses a new attribute from the version n entity type. Hence often you may find that the current production version of an object has had to be migrated to the backup model to support the migration of another object and the backup model does not therefore contain the previous version of the object.

Gen 8.0 FP1 and Batch z/OS Libraries

A previous posting described the new Gen 8.0 feature for z/OS dynamic RI triggers and libraries and an issue related to batch support. This has been addressed in Gen 8.0 Feature Pack 1 which is currently under beta test. FP1 also introduces some a useful enhancement to host encyclopaedia construction in allowing you to specify separate on-line and batch load libraries, so that when a dynamic action block needs to be linked separately for on-line and batch, this can be done from the same business system into separate load libraries.