Search

Entry Inspector – A Java FilterAPI Plugin Sample

Java Plugins are a great way to add new capabilities to BMC Remedy AR System without having to use the unpopular term ‘customization’. BMC have been adding more and more complex and sophisticated capabilities over last few releases by way of Java plugins so why can’t we?

In order to deploy Java Plugins on your system, you need to make changes to several areas of the installs so as a pre-requisit, you must have access to the server upon which AR System is installed. At a high level, the following is basic set of activities required to deploy a new Java Plugin.

  1. Develop your Plugin and package into a proper Jar file
  2. Deploy your Jar file in the pluginsvr directory
  3. Add an entry for your plugin in the pluginsvr_config.xml file
  4. Add an entry for your plugin in the ar.conf file

Now let’s look at the sample I have put together that satisfies an itch I’ve had since the move away from the User Tool in recent years…  As I’m sure most have realized already, pulling a report on a form with many fields (like HPD:Help Desk as an example) can be frustrating via the Mid-Tier.  If you are making an actual report, going through the effort is worth it but sometimes you just need to dump the data (including fields not on view) in order to see everything to support some customization that are in progress or not behaving correctly.  This scenario in Mid-Tier requires many clicks with new windows popping up and lots of frustrations (especially if it’s the first time you target a form as you need to build the report).  Back in the UserTool, it was much faster to just get a snapshot of the current data via reporting and this is something I miss.

Introducing the Entry Inspector plugin.  This plugin allows for quickly viewing the full contents of any entry from any form and then also give the ability to display only delta’s of that entry after a change has been made.  What this is useful for is you launch a custom Remedy form which I have included in the package that allows you to specify the Form and Entry to view and then the plugin will return all the data within the entry.  While you have the entry being displayed, if you go and make a change to the original entry, you can then come back and hit the Fetch Delta Updates button and it will show you all the fields that have now been updated which gives you a nice visibility into what specifically has changed to both visible and non-visible fields.

Here is a screenshot of this plugin in action via the provided Remedy form.

Entry Inspector

The sample above shows an Incident record loaded and then updates were made to the Impact, Urgency, Notes, Opt Cat 1-2-3 and Status.  As you can see, simply changing those values also changes other values on other fields so those are displayed as well as we are compared all fields to the previous Fetch of the entry record.  The right hand field will only ever contain elements that have changed from the left hand field.

The default field ID to search on for all forms is the Request ID Field (Field ID 1) however for some ITSM forms, this field is hidden and not easy to get to so to simplify things, I’ve added the ability to both define default Field ID to search on based on specific forms (in the sample above, HPD:Help Desk has a default ID to search on of the Incident Number field – 1000000161 rather then the Entry ID field – 1  ) as well as the ability to manually override the default field ID to search on.  This configuration is data driven based on another form called SA:EntryInspector_FormSearchConfig included in the definition file, simply create entries in there and they will be leveraged in the main Entry Inspector form.  Be aware though that whatever Field ID goes in there needs to be something that is guaranteed to be unique, like the request ID or a GUID.  There is a small amount of workflow required to make this form work but it is all self-contained and has no dependencies on any OOTB workflow or forms.

Now let’s look at the Java Plugin that makes the above possible and learn a little something about Java Plugins along the way.

In order to interact with Java Plugins,  specifically in this case Java FilterAPI Plugins which run server side, we need to leverage them as an property in a Set Fields action of a Filter.  This will require us to provide both input and output values.  The input values get sent to the plugin, the plugin does it’s work and then simply returns a List<Value> object that we can map into the output paramter.  This is the basic principal behind the plugins, you send it some data, it does it’s work and sends you the results.  Because we use Java for this, we can do pretty much anything we want as we are not limited by the boundaries of AR System workflow.

Full java source and AR definition file  will be provided at the bottom of this blog for the full example but here we will break things piece by piece.  Let’s start by understanding the basic structure of the Java FilterAPI plugin required:

The Java Plugin API is provided by way of the arpluginsvr81_build001.jar file (for v8.1 of the Plugin Server API, change to appropriate version for your environment).  Once you have loaded this library into your IDE, you will need to import a couple of Classes at a minimum:

import com.bmc.arsys.pluginsvr.plugins.ARFilterAPIPlugin;
import com.bmc.arsys.pluginsvr.plugins.ARPluginContext;

Now that we have those Classes loaded, we can create a stub to build out our functionality:

package net.soleauthority.arsys.plugin;

import com.bmc.arsys.api.ARException;
import com.bmc.arsys.api.ARServerUser;
import com.bmc.arsys.api.Value;
import com.bmc.arsys.pluginsvr.plugins.ARFilterAPIPlugin;
import com.bmc.arsys.pluginsvr.plugins.ARPluginContext;

import java.util.ArrayList;
import java.util.List;

public class SampleJavaFilterAPIPlugin extends ARFilterAPIPlugin {
    public List filterAPICall(ARPluginContext context, List paramList)
            throws ARException {

        ARServerUser ars = new ARServerUser(context, "", context.getARConfigEntry("Server-Name"));
        List results = new ArrayList();

        return results;
    }
}

The above is the bare minimum for being able to load as a valid Java plugin. It basically does nothing but will load and execute fine if called. You should note a couple of things, first we are not creating a net new context back to the AR System, we are using the existing context from the Java Plugin Server itself which is much quicker / recommended then creating a new context. You will also notice that we are creating a List<Value> object called results that must be returned from the plugin, this is what eventually will get returned into the Output mapping of the filter Set Fields action.  Specific to my sample form that goes along with this, you’ll notice that because I want this to be an interactive form, I am using a combination of active links that trigger service filters based upon the action I want to perform (FETCH or DELTA).

Now let’s add some code that actually does stuff, first let’s use the List<Value> paramList that we get when our plugin gets instanciated which is basically all the input values from our Filter Set Fields action.  First let’s look at what that Filter Set Fields looks like:

EntryInspector_SetFields

Above we can see that we need to pass in 5 different parameters and then expect a single value returned.  Based on this, let’s add to our plugin.


package net.soleauthority.arsys.plugin;

import com.bmc.arsys.api.ARException;
import com.bmc.arsys.api.ARServerUser;
import com.bmc.arsys.api.Value;
import com.bmc.arsys.pluginsvr.plugins.ARFilterAPIPlugin;
import com.bmc.arsys.pluginsvr.plugins.ARPluginContext;

import java.util.ArrayList;
import java.util.List;

public class SampleJavaFilterAPIPlugin extends ARFilterAPIPlugin {
    public List filterAPICall(ARPluginContext context, List paramList)
            throws ARException {

        ARServerUser ars = new ARServerUser(context, "", context.getARConfigEntry("Server-Name"));
        List results = new ArrayList();

        String targetSchema = paramList.get(0).getValue().toString(); // Pass it in the FormName
        String fieldID = paramList.get(1).getValue().toString(); // Pass in the FieldId to Search
        String searchValue = paramList.get(2).getValue().toString();  // Pass it in the Search Value
        String entryData = null;
        if (paramList.get(3).getValue() != null) {
            entryData = paramList.get(3).getValue().toString(); // Pass it in the Entry - empty on initial fetch
        }
        String action = paramList.get(4).getValue().toString(); // FETCH or DELTA

        return results;
    }
}

Here we can see that we take each of those parameters and map it into some usable variables for us. We are still not really doing anything with this data so now let’s take this the rest of the way and add our logic. We will see below that for the FETCH action that we will create a List object that has for each line a fieldName: FieldValue structure which will eventually be returned as a single Value added to our List object that must be returned from the plugin. For the DELTA action, we pass in both the previous entry we retrieved and then remove from our eventual results all the matching lines so that we are left with a List of only fields that have changed and we return that as our results to be populated into the right hand field of our Entry Inspector form.

/**
 * Copyright (c) 2014 Curtis Gallant <cgallant@soleauthority.net>
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

package net.soleauthority.arsys.plugin;

import com.bmc.arsys.api.ARException;
import com.bmc.arsys.api.ARServerUser;
import com.bmc.arsys.api.AttachmentField;
import com.bmc.arsys.api.AttachmentValue;
import com.bmc.arsys.api.CharacterField;
import com.bmc.arsys.api.Constants;
import com.bmc.arsys.api.CurrencyField;
import com.bmc.arsys.api.CurrencyValue;
import com.bmc.arsys.api.DateTimeField;
import com.bmc.arsys.api.DiaryField;
import com.bmc.arsys.api.DiaryItem;
import com.bmc.arsys.api.DiaryListValue;
import com.bmc.arsys.api.Entry;
import com.bmc.arsys.api.EntryListInfo;
import com.bmc.arsys.api.EnumItem;
import com.bmc.arsys.api.Field;
import com.bmc.arsys.api.FuncCurrencyInfo;
import com.bmc.arsys.api.QualifierInfo;
import com.bmc.arsys.api.RealField;
import com.bmc.arsys.api.RealFieldLimit;
import com.bmc.arsys.api.SelectionField;
import com.bmc.arsys.api.SelectionFieldLimit;
import com.bmc.arsys.api.StatusHistoryItem;
import com.bmc.arsys.api.StatusHistoryValue;
import com.bmc.arsys.api.Timestamp;
import com.bmc.arsys.api.Value;
import com.bmc.arsys.pluginsvr.plugins.ARFilterAPIPlugin;
import com.bmc.arsys.pluginsvr.plugins.ARPluginContext;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;

public class EntryInspector extends ARFilterAPIPlugin {
    public List filterAPICall(ARPluginContext context, List paramList)
            throws ARException {

        ARServerUser ars = new ARServerUser(context, "", context.getARConfigEntry("Server-Name"));
        List results = new ArrayList();

        String targetSchema = paramList.get(0).getValue().toString(); // Pass it in the FormName
        String fieldID = paramList.get(1).getValue().toString(); // Pass in the FieldId to Search
        String searchValue = paramList.get(2).getValue().toString();  // Pass it in the Search Value
        String entryData = null;
        if (paramList.get(3).getValue() != null) {
            entryData = paramList.get(3).getValue().toString(); // Pass it in the Entry - empty on initial fetch
        }
        String action = paramList.get(4).getValue().toString(); // FETCH or DELTA

        String qualStr = "'" + fieldID + "' = \"" + searchValue + "\"";
        QualifierInfo qual = ars.parseQualification(targetSchema, qualStr);
        List entries = ars.getListEntry(targetSchema, qual, 0, 1, null, null, false, null);
        List fields = ars.getListFieldObjects(targetSchema, Constants.AR_FIELD_TYPE_DATA);

        for (EntryListInfo entryListInfo : entries) {
            Entry entry = ars.getEntry(targetSchema, entryListInfo.getEntryID(), null);
            if (action.equals("FETCH")) {
                List entryStrings = prettyPrintEntry(entry, fields);
                results = createListOfValues(entryStrings);
            } else if (action.equals("DELTA")) {
                List originalEntry = convertStringToStringList(entryData);
                List updatedEntry = prettyPrintEntry(entry, fields);
                updatedEntry.removeAll(originalEntry);
                results = createListOfValues(updatedEntry);
            }
        }
        if (results.isEmpty()) {
            results.add(new Value("No Match Found"));
        }
        return results;
    }

    private List prettyPrintEntry(Entry entry, List fields) throws ARException {
        List recordOutput = new ArrayList();
        for (Field field : fields) {
            if (field.getFieldOption() != 4) {
                Integer fieldID = field.getFieldID();
                Value val = entry.get(fieldID);
                if (val.toString() != null) {
                    if (field instanceof DateTimeField) {
                        Timestamp callDateTimeTS = (Timestamp) val.getValue();
                        if (callDateTimeTS != null) {
                            recordOutput.add(field.getName() + ": " + callDateTimeTS.toDate().toString());
                        }
                    } else if (field instanceof SelectionField) {
                        SelectionFieldLimit selfieldLimit = (SelectionFieldLimit) field.getFieldLimit();
                        List eItemList = selfieldLimit.getValues();
                        for (EnumItem eItem : eItemList) {
                            if (eItem.getEnumItemNumber() == Integer.parseInt(val.toString()))
                                recordOutput.add(field.getName() + ": " + eItem.getEnumItemName());
                        }
                    } else if (field instanceof DiaryField) {
                        DiaryListValue dia = (DiaryListValue) val.getValue();
                        if (dia != null) {
                            StringBuilder diaBuilder = new StringBuilder();
                            for (DiaryItem dlv : dia) {
                                diaBuilder.append(dlv.getUser()).append(" ").append(dlv.getTimestamp().toDate().toString()).append(" ").append(dlv.getText()).append("  ");
                            }
                            recordOutput.add(field.getName() + ": " + diaBuilder);
                        }
                    } else if (field instanceof CurrencyField) {
                        CurrencyValue callCurrencyValue = (CurrencyValue) val.getValue();
                        if (callCurrencyValue != null) {
                            for (FuncCurrencyInfo currInfo : callCurrencyValue.getFuncCurrencyList()) {
                                recordOutput.add(field.getName() + ": " + currInfo.getValue() + " " + currInfo.getCurrencyCode());
                            }
                        }
                    } else if (field instanceof RealField) {
                        RealFieldLimit realLimit = (RealFieldLimit) field.getFieldLimit();
                        int realPrecisionLimit = realLimit.getPrecision();
                        recordOutput.add(field.getName() + ": " + String.format("%1$." + realPrecisionLimit + "f", val.getDoubleValue()));
                    } else if (field instanceof CharacterField) {
                        if (fieldID != 15) {
                            recordOutput.add(field.getName() + ": " + val.toString());
                        } else {
                            StatusHistoryValue shVal = StatusHistoryValue.decode(val.getValue().toString());
                            StringBuilder shBuilder = new StringBuilder();
                            if (shVal != null) {
                                for (StatusHistoryItem shItem : shVal) {
                                    if (shItem != null) {
                                        shBuilder.append(shItem.getTimestamp().getValue()).append("\u0004").append(shItem.getUser()).append("\u0003");
                                    } else {
                                        shBuilder.append("\u0003");
                                    }
                                }
                                recordOutput.add(field.getName() + ": " + shBuilder.toString());
                            }
                        }
                    } else if (field instanceof AttachmentField) {
                        AttachmentValue callAttachValue = (AttachmentValue) val.getValue();
                        if (callAttachValue != null) {
                            String attName = callAttachValue.getName();
                            // Get rid of the folder path if this was a previously exported and then re-imported attachment
                            int lastPos = attName.lastIndexOf('\\');
                            String attNameShort = (lastPos < 0) ? attName : attName.substring(lastPos + 1);
                            recordOutput.add(field.getName() + ": " + attNameShort);
                        }
                    } else {
                        recordOutput.add(field.getName() + ": " + val.toString());
                    }
                } else {
                    recordOutput.add(field.getName() + ": " + "");
                }
            }
        }
        Collections.sort(recordOutput);
        return recordOutput;
    }

    private List createListOfValues(List recordOutput) {
        List listOfValues = new ArrayList();
        StringBuilder tmp = new StringBuilder();
        for (String row : recordOutput) {
            tmp.append(row).append("\n");
        }
        listOfValues.add(new Value(tmp.toString()));
        return listOfValues;
    }

    private List convertStringToStringList(String items) {
        List list = new ArrayList();
        String[] listItems = items.split("\n");
        Collections.addAll(list, listItems);
        return list;
    }
}

And there we have it, a simple yet slightly useful Java FilterAPI plugin that I hope will be beneficial to people who like me get a bit frustrated with what should be a simple task of viewing all the data contained within a record without having to jump through hoops. I created this as a Java FilterAPI plugin is simply to highlight and teach some basics of the plugin architecture and provide a sample for people to learn from, to take full advantage of it requires a couple of forms and some workflow (also included in the package below).

Please read the INSTRUCTIONS.txt file included in the zip file for detailed steps on implementing this in your environment if you are interested.  If there are any bugs found or enhancements that you would like to see, please leave a comment and I will see about updating the package.

And as always, happy coding 🙂

Data Purging – Using the Java API to gracefully purge old data

Most of us have come across a system that has forms that are so bogged down with millions of records of old data that it becomes a performance nightmare to work with.  In BMC Remedy ITSM systems, this typically ends up being audit forms, log forms, etc.  In situations like this, there are several options that can be explored such as:

  • Deleting chunks of data by hand
    • More difficult since the UserTool has gone the way of the dodo bird
    • Very slow since all entries get loaded into your browser which slows it to a crawl and chews up a tone of memory
    • Easy to experience timeouts if you try and be overzealous with your chunks
  • Writing workflow to delete data
    • Escalation & Filter combos are probably the most common approach taken by admins and consultants to get ride of data in a more controlled and automated fashion however this means customization which can sometimes be held up in red-tape or if not properly built can go wrong and cause harm.
    • The more forms you want to target with this approach, the more possible customization required to support it.
  • Sledge-Hammer approach (SQL Statements)
    • The absolute fastest way to remove data as it bypasses all the application layers however depending on the forms being targeted, a deep understanding of the relationships that may touch the table as well as a good general understanding of the AR System database schema is required
    • Truncate is amazing when you can use it (e.g. 100% dump) as it will work almost instantaneously
  • API Program with some smarts built in
    • Can control pretty much all aspects of what needs to be done and will get you to where you need to go without having to resort to manual delete, customization or SQL wizardry approaches.
    • Portable and extendable to your hearts (and programming brains) content.

Now let’s look at a solution to this problem using the API.  First let’s set some goals for our program:

  1. Let’s say you want to keep only the last N days of data in a particular set of forms to satisfy audit requirements
  2. We also need to be able to base N days upon either the Create Date (Field ID 3) or the Modified Date (Field ID 6).
  3. We need to do this during production hours (24×7 system) so we need to minimize the impact to the server

With the above in mind, let’s look at how we can tackle this.  As with all my Remedy Java API code, I have a yaml configuration parser that I use to feed in my connection details as well as any required parameters / inputs.  It is purely optional however if you want to use the subsequent class further down without change you will need this, otherwise you are free to modify the subsequent class to not require this.  (***Note, you can download a copy of all the source files at the bottom of this post).

Here is the code for the ParseConfig class:

/*
 * Copyright (c) 2014 Curtis Gallant <cgallant@soleauthority.net>
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

package net.soleauthority.arsys.utils;

import org.yaml.snakeyaml.Yaml;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;

public class ParseConfig {
    private static String configServer = null;
    private static String configUser = null;
    private static String configPassword = null;
    private static int configPort = 0;
    private static int configRPC = 0;
    private static String configQueryType = null;
    private static int configDaysToKeep = 0;
    private static int configActionDelay = 0;
    private static int configSearchChunk = 0;
    private static ArrayList configListOfForms;

    public static String getConfigServer() {
        return configServer;
    }

    public static String getConfigUser() {
        return configUser;
    }

    public static String getConfigPassword() {
        return configPassword;
    }

    public static int getConfigPort() {
        return configPort;
    }

    public static int getConfigRPC() {
        return configRPC;
    }

    public static String getConfigQueryType() {
        return configQueryType;
    }

    public static int getConfigDaysToKeep() {
        return configDaysToKeep;
    }

    public static int getConfigActionDelay() {
        return configActionDelay;
    }

    public static int getConfigSearchChunk() {
        return configSearchChunk;
    }

    public static ArrayList getConfigListOfForms() {
        return configListOfForms;
    }

    protected static void readConfiguration(String configFileName) {
        try {
            InputStream input;
            if (configFileName.equals("")) {
                input = new FileInputStream(new File("PurgeDataConfig.yaml"));
            } else {
                input = new FileInputStream(new File(configFileName));
            }
            Yaml yaml = new Yaml();
            Map<String, ArrayList> config = (Map<String, ArrayList>) yaml.load(input);

            // Connection Info
            ArrayList connInfo = config.values().iterator().next();
            for (Object entry : connInfo) {
                // Server Name
                if (((HashMap) entry).get("ServerName") != null) {
                    configServer = ((HashMap) entry).get("ServerName").toString();
                }
                // Username
                if (((HashMap) entry).get("UserName") != null) {
                    configUser = ((HashMap) entry).get("UserName").toString();
                }
                // Password
                if (((HashMap) entry).get("Password") != null) {
                    configPassword = ((HashMap) entry).get("Password").toString();
                }
                // Port
                if (((HashMap) entry).get("Port") != null) {
                    configPort = Integer.parseInt(((HashMap) entry).get("Port").toString());
                }
                // RPC
                if (((HashMap) entry).get("RPC") != null) {
                    configRPC = Integer.parseInt(((HashMap) entry).get("RPC").toString());
                }
                // Type of Query, either Create Date or Modified Date
                if (((HashMap) entry).get("Query Type") != null) {
                    configQueryType = ((HashMap) entry).get("Query Type").toString();
                }
                // Days to keep
                if (((HashMap) entry).get("Days to keep") != null) {
                    configDaysToKeep = Integer.parseInt(((HashMap) entry).get("Days to keep").toString());
                }
                // Search Chunks
                if (((HashMap) entry).get("Search Chunk") != null) {
                    configSearchChunk = Integer.parseInt(((HashMap) entry).get("Search Chunk").toString());
                }
                // Time between delete actions
                if (((HashMap) entry).get("Action Delay") != null) {
                    configActionDelay = Integer.parseInt(((HashMap) entry).get("Action Delay").toString());
                }
            }

            // Forms list
            configListOfForms = config.get("Forms to purge");

            // Close our config file
            input.close();

        } catch (FileNotFoundException e) {
            System.out.println("Can't find " + configFileName);
            System.exit(1);
        } catch (IOException e) {
            System.out.println(e.getMessage());
            System.exit(1);
        }
    }
}

This will give us a yaml configuration file reader that will provide all our connection info, options for setting the query type, number of days of data to keep, artificial lag between delete operations to reduce impact to production servers (and help avoid timeouts), search chunk capability as well as create an ArrayList of all the forms we want to include in our purging. Below is a sample PurgeDataConfig.yaml file that will be read by the above that I used for testing, it contains some default values and a form I loaded with 500k records of varying data, change these to whatever form(s) you wish to purge:

# Place your connection information in the block below target AR Server
Connection Info:
 - ServerName: <ServerName>
   UserName: <UserName>
   Password: <Password>
   Port: <Port>
   RPC: <RPC>

   # Set the query to be either based on Create Date (Field ID 3) or Modified Date (Field ID 6)
   # Possible values are: Create Date or Modified Date
   Query Type: Create Date

   # Set the number of days to keep in the target forms, should be a number like 30
   Days to keep: 30

   # Enter an optional amount of milliseconds delay between deletes to be less impacting to server, e.g. 10
   Action Delay: 0

   # Set value for Search Chunk if you wish to limit searches in size, can be useful if no Max GetList is defined on the server
   # e.g. 5000 to break searches in chunks of no more then 5000 to not time out server when we generate the list of records to delete
   # this setting is ignored if Max Get List is defined unless the value is smaller then the Max Get List value.  Default value is 0.
   Search Chunk: 0

# Place the list of forms you wish to purge in the block below, format is "-"
# If your form has dash in the name, encapsulate it in double-quotes
Forms to purge:
 - "CG:TESTPURGE"

Now that we can parse our configuration file, here is the real Class that does all the work of purging data.  Here is the high level flow of the program based on my findings (not including boilerplate stuff):

  1. Grab the list of forms defined in the config file
  2. Process each form one at a time by creating a list of EntryListInfo objects that contain nothing but the EntryIDs of all the records that should be deleted based upon the number of days and type of query (based on Create Date or Modified Data)
  3. Once we have our list of records to delete, delete one by one and optionally introduce lag (in milliseconds) between each delete operation (this delay comes from the configuration file).

And now here is the code for our PurgeData class, reminder that you can download the code and a fully packed executable jar file if you want a copy of the code and the program without having to compile yourself.  You’ll notice I threw some additional goodies in there like a progress bar and per form timers, both are purely optional however it’s nice to give visual feedback while it’s running, especially if run against millions of records where it could be running for hours on end (or if you have put in a lot of delay between delete operations).

/*
 * Copyright (c) 2014 Curtis Gallant <cgallant@soleauthority.net>
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

package net.soleauthority.arsys.utils;

/**
 * PurgeData - This program will purge data from target forms based upon how many days you want to keep
 */

import com.bmc.arsys.api.ARException;
import com.bmc.arsys.api.ARServerUser;
import com.bmc.arsys.api.EntryListFieldInfo;
import com.bmc.arsys.api.EntryListInfo;
import com.bmc.arsys.api.OutputInteger;
import com.bmc.arsys.api.QualifierInfo;
import com.bmc.arsys.api.ServerInfoMap;

import java.util.ArrayList;
import java.util.Date;
import java.util.List;
import java.util.concurrent.TimeUnit;

public class PurgeData {
    private static ARServerUser ctx;

    private PurgeData(String configFileName) {
        ParseConfig.readConfiguration(configFileName);
        ctx = new ARServerUser();
        ctx.setServer(ParseConfig.getConfigServer());
        ctx.setUser(ParseConfig.getConfigUser());
        ctx.setPassword(ParseConfig.getConfigPassword());
        ctx.setPort(ParseConfig.getConfigPort());
        if (ParseConfig.getConfigRPC() != 0) {
            try {
                ctx.usePrivateRpcQueue(ParseConfig.getConfigRPC());
            } catch (ARException e) {
                System.out.println("Invalid RPC port specified for " + ParseConfig.getConfigServer() + ": " + ParseConfig.getConfigRPC());
            }
        }
    }

    public static void main(String[] args) {
        long startTime;
        PurgeData ars;
        if (args.length > 0) {
            ars = new PurgeData(args[0]);
        } else {
            ars = new PurgeData("");
        }
        ars.connectionTest();
        for (Object schema : ParseConfig.getConfigListOfForms()) {
            startTime = timerStart();
            processEntries(schema.toString(), ParseConfig.getConfigDaysToKeep(), ParseConfig.getConfigQueryType());
            timerStop(startTime);
        }
        ctx.logout();
    }

    private static void processEntries(String schemaName, int daysToKeep, String queryType) {
        List<EntryListInfo> entryLists;
        String qualString = null;
        QualifierInfo qual;
        if (queryType.equals("Create Date")) {
            qualString = "'3' >= ($TIMESTAMP$ - (60*60*24*" + daysToKeep + "))";
        } else if (queryType.equals("Modified Date")) {
            qualString = "'6' >= ($TIMESTAMP$ - (60*60*24*" + daysToKeep + "))";
        }
        try {
            qual = ctx.parseQualification(schemaName, qualString);
            int maxGetList = getMaxGetListForServer();
            if (ParseConfig.getConfigSearchChunk() > 0 && ParseConfig.getConfigSearchChunk() < maxGetList) {
                entryLists = getEntryListInfos(schemaName, qual, ParseConfig.getConfigSearchChunk());
            } else {
                entryLists = getEntryListInfos(schemaName, qual, maxGetList);
            }
            System.out.println("\nDeleting " + entryLists.size() + " total entries from schema: " + schemaName);
            int counter = 0;
            for (EntryListInfo entryList : entryLists) {
                ctx.deleteEntry(schemaName, entryList.getEntryID(), 0);
                Thread.sleep(ParseConfig.getConfigActionDelay());
                counter++;
                printProgressBar((counter * 100) / entryLists.size());
            }
            System.out.println("\n");
        } catch (ARException e) {
            System.out.println("Can't get list of entries for deleting: " + e.getMessage());
            System.exit(1);
        } catch (InterruptedException e) {
            System.out.println(e.getMessage());
        }
    }

    private static int getMaxGetListForServer() throws ARException {
        ServerInfoMap maxGetListConfiguration = ctx.getServerInfo(new int[]{28}); // 28 = MaxGetList
        return Integer.parseInt(maxGetListConfiguration.firstEntry().getValue().toString());
    }

    private static List<EntryListInfo> getEntryListInfos(String schemaName, QualifierInfo qual, int maxResults) throws ARException {
        OutputInteger totalCount = new OutputInteger();
        List<EntryListInfo> entryLists;
        List<EntryListFieldInfo> returnFields = new ArrayList<EntryListFieldInfo>();
        EntryListFieldInfo returnField = new EntryListFieldInfo(1);
        returnFields.add(returnField);
        entryLists = ctx.getListEntry(schemaName, qual, 0, maxResults, null, returnFields, true, totalCount);
        // If there are more results then maxResults will allow to fetch, iterate over until we have our fully populated List
        if (entryLists.size() < totalCount.intValue()) {
            for (int i = entryLists.size(); i <= totalCount.intValue(); i += maxResults) {
                entryLists.addAll(ctx.getListEntry(schemaName, qual, i, maxResults, null, returnFields, true, null));
            }
        }
        return entryLists;
    }

    private static void printProgressBar(int percent) {
        StringBuilder bar = new StringBuilder("[");
        for (int i = 0; i < 50; i++) {
            if (i < (percent / 2)) {
                bar.append("=");
            } else if (i == (percent / 2)) {
                bar.append(">");
            } else {
                bar.append(" ");
            }
        }
        bar.append("]   ").append(percent).append("%     ");
        System.out.print("\r" + bar.toString());
    }

    private static long timerStart() {
        return new Date().getTime();
    }

    private static void timerStop(long lStartTime) {
        long lEndTime = new Date().getTime();
        long difference = lEndTime - lStartTime;
        System.out.println("Elapsed time: " + String.format("%d min, %d sec",
                TimeUnit.MILLISECONDS.toMinutes(difference),
                TimeUnit.MILLISECONDS.toSeconds(difference) - TimeUnit.MINUTES.toSeconds(TimeUnit.MILLISECONDS.toMinutes(difference))));
    }

    private void connectionTest() {
        System.out.println();
        try {
            ctx.verifyUser();
        } catch (ARException e) {
            System.out.println("Could not log into server: " + e);
            System.exit(1);
        }
    }
}

There you have it, a sample POC for purging large amounts of data from a Remedy system. This utility is meant to be a proof of concept and while it works well for me, it has not been heavily tested so use with caution. If you have a very fast system, there really is no need to introduce lag with the Action Delay or chunks so just keep both at zero. If your Remedy system on occasion or on large operations typically times out, I’d recommend starting with say 10 (milliseconds) in the Action Delay option and 2500 for the Search Chunk and see how it goes. If timeouts continue to occur (during the delete actions), you can slow creep up the Action Delay in 5ms increments, if it times out in gathering the list of entries to delete, you can lower the Chunk Size until you find the sweet spot for your server.

It should be noted that you can run this program over and over again or schedule it to run on a periodic basis if you wanted it to.  Since the premise of this sample program is to keep data that is N days old, you can run it as many times as you need to get to your desired number of entries and each iterative time it will get faster and faster as the raw counts drop to the point that running it on a daily basis may run in a matter of seconds instead of big hours long runs deleting millions of records each time as they pile up over months/years again. Functionality could be added to make this more dynamic like being able to write your own qualification in the configuration file for maximum flexibility however to keep it simple and to satisfy our stated requirements I’ve only let the number of days be the quantitative value in the control of the user.

It should also be noted again that this is a sample program and doesn’t go all out in proper exception handling so keep that in mind and treat it as a simple utility, not a bulletproof solution.

I hope this can be useful to somebody and as always happy coding 🙂

{wpdm_category=code}

Replicating “Save As” functionality of Developer Studio using Java API

Yesterday there was a BMC Community question around replicating the ‘Save As’ functionality of Developer Studio for Forms using only the Java API.  This ends up being more complicated then it seems so I decided to take a stab yesterday evening at putting together a demonstration program that would accomplish this goal (for Forms only at this point).

Some background; In order to make a true copy of a Form, you need to also make copies of the View and Field objects.  This is complicated by the fact that for more complex forms, there is the issue with field dependencies, e.g. you can’t create a column field until the table holder is created, you can’t create a panel until the panel holder is created, you can’t add a simple character field that lives on a panel until the panel and panel holder is created, etc.  So you need to bring in the fields one at a time by type (unless there is some other method of doing this).

Here is an example of what I’m talking about:

nestedFields
Now let’s look at some code that can solve this mess…

As with all my Remedy Java API code, I have a yaml configuration parser that I use to feed in my connection details as well as any required parameters / inputs.  It is purely optional however if you want to use the subsequent class further down without change you will need this, otherwise you are free to modify the subsequent class to not require this.  (***Note, you can download a copy of all the source files at the bottom of this post)

Here is the code to that class:

/*
 * Copyright (c) 2013 Curtis Gallant <cgallant@gmail.com>
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

package net.soleauthority.arsys.utils;

import org.yaml.snakeyaml.Yaml;

import java.io.File;
import java.io.FileInputStream;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.HashMap;
import java.util.Map;

class ParseConfig {
    private static String configServer = null;
    private static String configUser = null;
    private static String configPassword = null;
    private static int configPort = 0;
    private static String configPrefix = null;
    private static ArrayList configListOfFormsToCopy;

    public static String getConfigServer() { return configServer; }
    public static String getConfigUser() { return configUser; }
    public static String getConfigPassword() { return configPassword; }
    public static int getConfigPort() { return configPort; }
    public static String getConfigPrefix() { return configPrefix; }
    public static ArrayList getConfigListOfFormsToCopy() { return configListOfFormsToCopy;}

    static void readConfiguration() {
        try {
            InputStream input = new FileInputStream(new File("CloneRemedyObjectConfig.yaml"));
            Yaml yaml = new Yaml();
            Map<String, ArrayList> config = (Map<String, ArrayList>) yaml.load(input);

            // Connection Info
            ArrayList connInfo = config.values().iterator().next();
            for (Object entry : connInfo) {
                // Server Name
                if (((HashMap) entry).get("Server Name") != null) {
                    configServer = ((HashMap) entry).get("Server Name").toString();
                }
                // Username
                if (((HashMap) entry).get("User Name") != null) {
                    configUser = ((HashMap) entry).get("User Name").toString();
                }
                // Password
                if (((HashMap) entry).get("Password") != null) {
                    configPassword = ((HashMap) entry).get("Password").toString();
                }
                // Port
                if (((HashMap) entry).get("Port") != null) {
                    configPort = Integer.parseInt(((HashMap) entry).get("Port").toString());
                }
                // Prefix
                if (((HashMap) entry).get("Prefix") != null) {
                    configPrefix = ((HashMap) entry).get("Prefix").toString();
                }
            }

            // Forms to purge list
            configListOfFormsToCopy = config.get("Forms");

            // Close our config file
            input.close();

        } catch (FileNotFoundException e) {
            System.out.println("Can't find CloneRemedyObjectConfig.yaml file");
            System.exit(1);
        } catch (IOException e) {
            System.out.println(e.getMessage());
            System.exit(1);
        }
    }
}

This will give us a yaml configuration file reader that will provide all our connection info, prefix for newly created forms as well as create an ArrayList of all the forms we want to make copies of. Below is a sample CloneRemedyObjectConfig.yaml file that will be read by the above that I used for testing, it contains one of each of the Form types I chose at random, change these to whatever form you wish to copy:

# Place your connection information in the block below target AR Server
Connection Info:
 - Server Name: <serverName>
   User Name: <adminLoginName>
   Password: <adminLoginPassword>
   Port: <serverPort>

   # Append the following prefix for newly copied forms, make sure to encapsulate with double-quotes
   Prefix: "CG:"

# Place the list of forms you wish to make copies, format is <space>-<space>"<formName>"
# Don't forget the double-quotes
Forms:
- "AST:SoftwareUsageDialog"            #Display form
- "AST:SoftwareUsage"                  #Vendor form
- "AST:Statistics"                     #Regular form
- "AST:SoftwareServer"                 #Join form
- "AR System Metadata: actlink_macro"  #View form

Now that we can parse our configuration file, here is the real Class that does all the work of copying a form.  Here is the high level flow of the program based on my findings (not including boilerplate stuff):

  1. Check if the the new form name will already exist and if it does skip it entirely
  2. If it doesn’t exist, find out what type of form the original one is (Regular, DisplayOnly, Vendor, View or Join) and take appropriate action depending on the form
    • Because when we clone our form, initially the fields/views do not exist so things like results list, indexes, audits and archive settings will throw errors if you just try and createForm the cloned object so we need to dump some properties them prior to our base form creation
    • We then set some of the basics like the new name, owner, etc.
    • If the original form was part of an Application, we need to remove that (same behavior as Developer Studio)
  3. Create the basic form based on our modified clone
  4. Make copies of the View objects from the original form, taking into account that if the form used the Default Administrator View, it will collide with the new view we are trying to create so we need to wrap our attempt at createView to create the view if it truly is new, or if the name/ID is the same as the default on newly created forms, do a setView instead.
  5. Now that we have our views in place we need to copy all the fields, first we get a list of all the fields from the original form and then need to split them into separate lists based on the type of field it is
    • We set the new target form name for the fields as well as set the ReserverID flag to true so that we don’t get errors creating fields in the BMC reserved range
    • For table fields specifically, we need to first strip out the qualification from its TableFieldLimit since it could possibly reference fields that haven’t been created yet (e.g. other table fields or columns)
  6. We then can create all the fields based on our new lists, the order I found that works is the following based on possible hierarchy dependencies for various field properties:
    • PageHolderFields
    • PageFields
    • TrimFields
    • DataFields – these are pretty much every real field on a form other then core form fields
    • CoreFields – these are core fields like 1-8, 15, etc.
      • For some form types we need a bit more logic, e.g. Vendor/Join forms by default create fieldID 1 so for those forms we can’t simply include those fields in the createMultipleFields call, we need to update each of those fields individually with either setField if they already exist so that those already created fields will receive the changes from the origional one (e.g. DisplayProperties) or createField for the remainder of the core fields
    • TableFields
      • Before we create the table fields, now that the fields above have been created, we can re-add the qualification to the object and then create.
    • ColumnFields
    • AttachmentPoolFields
    • AttachmentFields
    • ControlFields

I’ve only added a few scattered comments throughout to help understand some of the code however I have not really documented to much at this point. If there is anything unclear, please let me know in the comments. If you find any bugs also please let me know in the comments. Again this was simply built as a POC and has not been heavily tested outside of my own system and about 30 various forms (of different types). I’ve also left most of the exception handling throw the stack out so that if there are real bugs I can get the details of where it fails, any real program based on this should do exceptions in a ‘nicer’ manner.

/*
 * Copyright (c) 2013 Curtis Gallant <cgallant@gmail.com>
 *
 * Permission to use, copy, modify, and distribute this software for any
 * purpose with or without fee is hereby granted, provided that the above
 * copyright notice and this permission notice appear in all copies.
 *
 * THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES
 * WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF
 * MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR
 * ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
 * ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF
 * OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 */

package net.soleauthority.arsys.utils;

/**
 * CloneRemedyObject - This sample program will create a copy of an AR Form with a defined prefix
 */

import com.bmc.arsys.api.ARException;
import com.bmc.arsys.api.ARServerUser;
import com.bmc.arsys.api.AttachmentField;
import com.bmc.arsys.api.AttachmentPoolField;
import com.bmc.arsys.api.ColumnField;
import com.bmc.arsys.api.Constants;
import com.bmc.arsys.api.ControlField;
import com.bmc.arsys.api.DisplayOnlyForm;
import com.bmc.arsys.api.Field;
import com.bmc.arsys.api.Form;
import com.bmc.arsys.api.JoinForm;
import com.bmc.arsys.api.ObjectPropertyMap;
import com.bmc.arsys.api.PageField;
import com.bmc.arsys.api.PageHolderField;
import com.bmc.arsys.api.QualifierInfo;
import com.bmc.arsys.api.RegularForm;
import com.bmc.arsys.api.TableField;
import com.bmc.arsys.api.TableFieldLimit;
import com.bmc.arsys.api.TrimField;
import com.bmc.arsys.api.Value;
import com.bmc.arsys.api.VendorForm;
import com.bmc.arsys.api.View;
import com.bmc.arsys.api.ViewCriteria;
import com.bmc.arsys.api.ViewForm;

import java.util.ArrayList;
import java.util.HashMap;
import java.util.Iterator;
import java.util.List;
import java.util.Map;

public class CloneRemedyObject {
    private static ARServerUser ctx;

    public CloneRemedyObject() {
        ParseConfig.readConfiguration();
        ctx = new ARServerUser();
        ctx.setServer(ParseConfig.getConfigServer());
        ctx.setUser(ParseConfig.getConfigUser());
        ctx.setPassword(ParseConfig.getConfigPassword());
        ctx.setPort(ParseConfig.getConfigPort());
    }

    public static void main(String[] args) {
        CloneRemedyObject ars = new CloneRemedyObject();
        ars.connectionTest();
        try {
            ctx.useAdminRpcQueue();
            // We will want to create our form as a custom object so set in Best Practice Mode
            ctx.setBaseOverlayFlag(false);
            ctx.setOverlayFlag(true);
            ctx.setOverlayGroup(String.valueOf(Constants.AR_GROUP_ID_ADMINISTRATOR));
            ctx.setDesignOverlayGroup(String.valueOf(Constants.AR_GROUP_ID_ADMINISTRATOR));

            for (Object originalFormName : ParseConfig.getConfigListOfFormsToCopy()) {
                copyForm(originalFormName.toString(), ParseConfig.getConfigPrefix());
            }
        } catch (ARException e) {
            e.printStackTrace();
        }
        ctx.logout();
    }

    private static void copyForm(String originalFormName, String newFormPrefix) {
        try {
            List<String> existingCheck = ctx.getListForm();
            if (existingCheck.contains(newFormPrefix + originalFormName)) {
                System.out.println("Target form already exists, skipping");
                return;
            }
            Form originalForm = ctx.getForm(originalFormName);
            String formType = String.valueOf(originalForm.getClass());
            Form newForm = null;
            if (formType.equals("class com.bmc.arsys.api.RegularForm")) {
                newForm = (RegularForm) originalForm.clone();
                newForm.setSortInfo(null);
                newForm.setAuditInfo(null);
                newForm.setArchiveInfo(null);
                newForm.setEntryListFieldInfo(null);
                newForm.setIndexInfo(null);
            } else if (formType.equals("class com.bmc.arsys.api.DisplayOnlyForm")) {
                newForm = (DisplayOnlyForm) originalForm.clone();
            } else if (formType.equals("class com.bmc.arsys.api.JoinForm")) {
                newForm = (JoinForm) originalForm.clone();
                newForm.setSortInfo(null);
                newForm.setAuditInfo(null);
                newForm.setArchiveInfo(null);
                newForm.setEntryListFieldInfo(null);
            } else if (formType.equals("class com.bmc.arsys.api.ViewForm")) {
                newForm = (ViewForm) originalForm.clone();
                newForm.setSortInfo(null);
                newForm.setAuditInfo(null);
                newForm.setArchiveInfo(null);
                newForm.setEntryListFieldInfo(null);
                newForm.setIndexInfo(null);
            } else if (formType.equals("class com.bmc.arsys.api.VendorForm")) {
                newForm = (VendorForm) originalForm.clone();
                newForm.setSortInfo(null);
                newForm.setAuditInfo(null);
                newForm.setArchiveInfo(null);
                newForm.setEntryListFieldInfo(null);
                newForm.setIndexInfo(null);
            }

            // Set some of the defaults from original form
            assert newForm != null;
            newForm.setName(newFormPrefix + originalFormName);
            newForm.setDefaultVUI("Change Me"); // In-case the original form uses default name
            newForm.setOwner(originalForm.getOwner());
            newForm.setHelpText(originalForm.getHelpText());
            newForm.setDiary(originalForm.getDiary());
            newForm.setLastChangedBy(originalForm.getLastChangedBy());
            newForm.setPermissions(originalForm.getPermissions());

            // Set Object Property Map after removing app owner and deployable application tags
            ObjectPropertyMap propertyMap = (ObjectPropertyMap) originalForm.getProperties().clone();
            for (Iterator<Map.Entry<Integer, Value>> it = propertyMap.entrySet().iterator(); it.hasNext(); ) {
                Map.Entry<Integer, Value> entry = it.next();
                if (entry.getKey().equals(Constants.AR_SMOPROP_APP_OWNER)
                        || entry.getKey().equals(Constants.AR_SMOPROP_APP_LIC_VERSION)
                        || entry.getKey().equals(Constants.AR_SMOPROP_APP_LIC_DESCRIPTOR)
                        || entry.getKey().equals(Constants.AR_SMOPROP_APP_LIC_USER_LICENSABLE)) {
                    it.remove();
                }
            }
            newForm.setProperties(propertyMap);

            // Attempt to create the form
            ctx.createForm(newForm);

            copyFormViews(originalFormName, newFormPrefix);
            copyFormFields(originalFormName, newFormPrefix, formType);

            // Set remaining form settings now that fields are created - Ignoring audit and archive settings
            Form updateForm = ctx.getForm(newFormPrefix + originalFormName);
            if (formType.equals("class com.bmc.arsys.api.RegularForm")) {
                updateForm.setEntryListFieldInfo(originalForm.getEntryListFieldInfo());
                updateForm.setSortInfo(originalForm.getSortInfo());
                updateForm.setIndexInfo(originalForm.getIndexInfo());
                ctx.setForm(updateForm);
            } else if (formType.equals("class com.bmc.arsys.api.JoinForm")) {
                updateForm.setEntryListFieldInfo(originalForm.getEntryListFieldInfo());
                updateForm.setSortInfo(originalForm.getSortInfo());
                ctx.setForm(updateForm);
            } else if (formType.equals("class com.bmc.arsys.api.VendorForm")) {
                updateForm.setEntryListFieldInfo(originalForm.getEntryListFieldInfo());
                updateForm.setSortInfo(originalForm.getSortInfo());
                ctx.setForm(updateForm);
            }
        } catch (ARException e) {
            e.printStackTrace();
        } catch (CloneNotSupportedException e) {
            e.printStackTrace();
        }
    }

    private static void copyFormViews(String originalFormName, String newFormPrefix) {
        try {
            ViewCriteria viewCriteria = new ViewCriteria();
            viewCriteria.setRetrieveAll(true);

            // Grab the Default Created View ID from Form copy so that we can delete the view later
            int deleteOrUpdateViewID = 0;
            List<View> deleteViews = ctx.getListViewObjects(newFormPrefix + originalFormName, 0,
                    viewCriteria);
            for (View deleteOrUpdateView : deleteViews) {
                deleteOrUpdateViewID = deleteOrUpdateView.getVUIId();
            }

            // Copy or Update the views from the original to the copy
            boolean keepView = false;
            List<View> views = ctx.getListViewObjects(originalFormName, 0, viewCriteria);
            for (View view : views) {
                View copiedView = (View) view.clone();
                copiedView.setFormName(newFormPrefix + originalFormName);
                if (copiedView.getVUIId() == deleteOrUpdateViewID) {
                    ctx.setView(copiedView);
                    keepView = true;
                } else {
                    ctx.createView(copiedView);
                }
            }
            // Delete the default view that got created since it's not on origional
            if (!keepView) {
                ctx.deleteView(newFormPrefix + originalFormName, deleteOrUpdateViewID);
            }
        } catch (ARException e) {
            e.printStackTrace();
        } catch (CloneNotSupportedException e) {
            e.printStackTrace();
        }
    }

    private static void copyFormFields(String originalFormName, String newFormPrefix, String formType) {
        List<Field> pageHolderFields = new ArrayList<Field>();
        List<Field> pageFields = new ArrayList<Field>();
        Map<Field, QualifierInfo> tableFields = new HashMap<Field, QualifierInfo>();
        List<Field> columnFields = new ArrayList<Field>();
        List<Field> attachPoolFields = new ArrayList<Field>();
        List<Field> attachFields = new ArrayList<Field>();
        List<Field> controlFields = new ArrayList<Field>();
        List<Field> trimFields = new ArrayList<Field>();
        List<Field> dataFields = new ArrayList<Field>();
        List<Field> coreFields = new ArrayList<Field>();

        try {
            List<Field> fields = ctx.getListFieldObjects(originalFormName);
            for (Field field : fields) {
                Field copiedField = (Field) field.clone();
                copiedField.setForm(newFormPrefix + originalFormName);
                copiedField.setReservedIDOK(true);
                if (!copiedField.isCoreField()) {
                    if (copiedField instanceof PageHolderField) {
                        pageHolderFields.add(copiedField);
                    } else if (copiedField instanceof PageField) {
                        pageFields.add(copiedField);
                    } else if (copiedField instanceof TableField) {
                        // For table fields, we must strip the qualification first and re-add later
                        TableFieldLimit tablefieldLimit = (TableFieldLimit) copiedField.getFieldLimit();
                        tablefieldLimit.setQualifier(null);
                        copiedField.setFieldLimit(tablefieldLimit);
                        tableFields.put(copiedField, ((TableFieldLimit) field.getFieldLimit()).getQualifier());
                    } else if (copiedField instanceof ColumnField) {
                        columnFields.add(copiedField);
                    } else if (copiedField instanceof AttachmentPoolField) {
                        attachPoolFields.add(copiedField);
                    } else if (copiedField instanceof AttachmentField) {
                        attachFields.add(copiedField);
                    } else if (copiedField instanceof ControlField) {
                        controlFields.add(copiedField);
                    } else if (copiedField instanceof TrimField) {
                        trimFields.add(copiedField);
                    } else {
                        dataFields.add(copiedField);
                    }
                } else {
                    coreFields.add(copiedField);
                }
            }
            if (pageHolderFields.size() > 0)
                ctx.createMultipleFields(pageHolderFields);
            if (pageFields.size() > 0)
                ctx.createMultipleFields(pageFields);
            if (trimFields.size() > 0)
                ctx.createMultipleFields(trimFields);
            if (dataFields.size() > 0)
                ctx.createMultipleFields(dataFields);
            if (coreFields.size() > 0) {
                if (formType.equals("class com.bmc.arsys.api.VendorForm")) {
                    for (Field field : coreFields) {
                        if (field.getFieldID() == 1) {
                            ctx.setField(field);
                        } else {
                            ctx.createField(field, true);
                        }
                    }
                } else if (formType.equals("class com.bmc.arsys.api.JoinForm")) {
                    for (Field field : coreFields) {
                        if (field.getFieldID() == 1 || field.getFieldID() == 15) {
                            ctx.setField(field);
                        } else {
                            ctx.createField(field, true);
                        }
                    }
                } else {
                    ctx.setMultipleFields(coreFields);
                }
            }
            if (tableFields.size() > 0) {
                for (Map.Entry<Field, QualifierInfo> tableField : tableFields.entrySet()) {
                    ctx.createField(tableField.getKey(), true);
                }
                if (columnFields.size() > 0) {
                    ctx.createMultipleFields(columnFields);
                }
                for (Map.Entry<Field, QualifierInfo> tableField : tableFields.entrySet()) {
                    TableFieldLimit tableFieldLimit = (TableFieldLimit) tableField.getKey().getFieldLimit();
                    tableFieldLimit.setQualifier(tableField.getValue());
                    tableField.getKey().setFieldLimit(tableFieldLimit);
                    ctx.setField(tableField.getKey());
                }
            }
            if (attachPoolFields.size() > 0)
                ctx.createMultipleFields(attachPoolFields);
            if (attachFields.size() > 0)
                ctx.createMultipleFields(attachFields);
            if (controlFields.size() > 0)
                ctx.createMultipleFields(controlFields);
        } catch (ARException e) {
            e.printStackTrace();
        } catch (CloneNotSupportedException e) {
            e.printStackTrace();
        }
    }

    private void connectionTest() {
        System.out.println();
        try {
            ctx.verifyUser();
        } catch (ARException e) {
            System.out.println("Could not log into server: " + e);
            System.exit(1);
        }
    }
}

Happy Coding 🙂