How to unlock a Datastage job

When a user is compiling a Datastage job,They are getting below message
This job is locked by user username.

Even if user has closed that job,this error is still coming.

OR

User aborted a running sequence in between and then getting below error while trying to compile the sequence again.

Cannot get exclusive access to executable file for job Job name-Job may be being monitored.


Solution:
1.Click on Tools and then Run Director from designer.
2.Select the job/Sequence and click on job from top and choose cleanup resources.

3.If this option is not available or visible to you,then open Datastage administrator.
4.Select that project and go to Properties.Please check marked checkbox.

5. Now click on the clean up resource from director.
6.Now try to compile the job to see if the issue is fixed.
7.If the issue still persist,Then click the session of that user from Infosphere.
8.Click on administration.
9.Click on Session management.
10.Click on Active session and select that user and click on Disconnect.

How to remove security alerts from Datastage client window

When we are trying to login into Datastage designer from client machine getting below security Popup.When we click on Yes,It disappears and we are able to login into Datastage designer.

But we try to run multiple projects simultaneously,It is really annoying.Please follow the below steps to get ride of this Popup.It will disappear.

Solution:

Please follow below steps.

1.To view the security certificate, Click on "View Certificate".
2.Click the "Certification Path" tab, and then select the "Root Certificate".
3.Click on "View Certificate".
4.Click on "Install Certificate", and then click "Next".
5.Select "Place all certificates in the following store".
6.Click "Browse", and then select "Trusted Root Certification Authorities".
7.Click "Next", and then click "Finish" to import the certificate.
Now the security Popup will disappear.

DataStage Job Compilation error "Failed to invoke GenRuntime using phantom process helper".

 
           

 Data Stage compilation error "Failed to invoke GenRuntime using phantom process helper. "

If you are getting below error while compiling jobs in Data Stage and error in coming in all the projects.Job are not able to execute.

Failed to invoke GenRuntime using phantom process helper. 

Cause:One of the possible cause of this error may be that tmp directory is full.Go to location
            /opt/IBM/InformationServer and check the space occupied by folder tmp

Run below command to see the space occupied.

df -k /opt/IBM/InformationServer/tmp

It command will show you total space assigned to this folder,space occupied,space available and %use.

If your issue is due to space in tmp directory,You will see %use as 100%.That means there is no space available in tmp directory and your jobs are failing.

If you try to delete some unwanted files from tmp folder,%Use may not decrease less than 100%.It means that even after deleting the files,no space is being released.

Solution:

Data Stage restart may resolve this issue.

Your tmp folder  may be filled because of the bad jobs.

A bad jobs is a job that had an issue while running and caused the temp directory to become full.
This could happen again if the same program is run.

Developer needs to fix the job to avoid this issue.

Data Stage Jobs are hanging after Upgrade/Installation

           

               Data stage jobs are hanging on new server (after upgrade)

 If you are facing below error-

After Data Stage upgrade or new installation, if your jobs are hanging during testing, hundreds of jobs run together but don’t get executed and get hanged. Jobs are taking a lot of time in execution.

This problem may be because the nproc value and max user processes variables don’t have enough values to execute the jobs.

[root@localhost# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127358
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65535
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

A decent value for max user processes should be 10K.Please check above values for dsadm account or the account which is running your Data stage server.

If open files and max user processes values are less then and get it fixed by Linux team.

Nproc values are saved in /etc/security/limits.conf .

If you are upgrading on a new server,You can verify these settings on your current server also.

DataStage configuration (WebSphere and InfoSphere)

             
                     

                  How to configure DataStage after installation

     We need to configure datastage once the installation is completed.In the previous posts,We have discussed that you need to copy contents from dsenv on your current server to your new server and some of the other configuration files.

After copying the configuration files,We need to configure datastage .Below are the steps to configure datastage.

First,we need to configure security in WebSphere.

                                  WebSphere configuration

1. Login into WebSphere admin URL  https://hostname:9043/ibm/console/login.do  with existing admin user( Which you have provided during installation ex wasadmin)

2. Click on Security---> global security.

3. Choose standalone ldap registry in the pull down window for ‘Available realm definitions’ then click configure.

    Make sure Enable administrative security is checked in administrative secuirty.

    Enable application security is checked in Application security.

          On the right side LTPA should be selected in Authentication.

   

4. Provide primary administrative user name(It should be LDAP account)

5. Choose Microsoft Active Directory in Types of LDAP server.

6. Provide Domain.corp.dom in host field and port number in port field.

7. You can keep the search time out same as default.

8. In the Bind Distinguished Name (DN)

Bind dn: cn=adminid_from_AD,ou=users,dc=corp,dc=dom

(You can check these settings in your old environment.)

9. In the Bind password,Provide the password of the above id ,which you have mentioned in DN.

10. At the top hits save.

11.   Go to Administrative group roles, hit add, change the search string to the name of the group that will be WAS admins and click add, then save(optional).

12. Click apply and then save.

After making these changes,You can see all the LDAP groups and users through IIS URL.

Now,you can login into WebSphere admin console,only with the account which you have provided in primary administrative user name.

If you want to login with some other accounts also to monitor the activities,then you need to add Monitor Roles for all Authenticated users.

1. Click on Administrative group roles under Users and Groups.

2. Click on Add.

3. In the group,Choose ALL AUTHENTICATED

4. In the Roles,Choose monitor.

5.Click on Apply.

6. Now all the users,who have access to datastage,should be able to login into WebSphere admin console with their user id.

                           InfoSphere Configuration

Credential Mapping

1) Defining default credentials

1. Open the admin console URL https://servername:9445/ibm/iis/console/.

2. Login with the user id which you gave during installation (Example wasadmin).

    You must have suite administrator authority or IBM® InfoSphere® DataStage® and IBM      InfoSphere QualityStage® administrator authority.

The default credentials are used for any users who do not have their own credential mappings. If you do not want users who do not have mapped credentials to access the server, do not add default mapping credentials.

Procedure

1. In the IBM InfoSphere Information Server Web console, click the Administration tab.
2. In the Navigator pane, select Domain Management > Engine Credentials.
3. Select the InfoSphere Information Server engine for which you want to specify the default credentials.
4. Click Open Configuration.
5. In the User Name field, type the user name to be used by all InfoSphere Information Server users for whom a specific mapping is not defined.
6. In the Password field, type the corresponding password. The user name and password that you provide must be a valid user name and password for the operating system where the engine tier components are installed.
7. Confirm the password.
8. Click Save and Close.

2.) Configuring your credentials

 you can map the credentials for your own user account.

Steps

1. In the  Information Server Web console, click the Administration tab.
2. In the Navigation pane, select Domain Management > Engine Credentials.
3. Select the InfoSphere Information Server engine that you want to configure.
4. Click Open My Credentials.
5. Type the user name and password that you want to use to connect to the IBM InfoSphere Information Server engine. The user name and password that you provide must be a valid user name and password for the operating system where the engine tier components are installed.
6. Click Save and Close.

3.) Grant role to DataStage users/Groups

Steps

1. Grant the required suite and suite component roles to the user in the Web console.
2. Using a role that has administrative privileges, log in to the IBM InfoSphere Information Server Web console.
3. Select the Administration tab.
4. In the Navigation pane, select Users and Groups > Users.
5. Select the user that you want to grant access to and click Open User.
6. In the Roles pane, assign the following roles to the user.
7. Suite User
8. Required for all users in order to log in to any of the suite components.
9. DataStage and QualityStage User
10. Required for any user in order log in to any of the InfoSphere DataStage and InfoSphere QualityStage product modules.
11. DataStage and QualityStage Administrator
12. Optional. Grants full access to all projects and the administrative capability of InfoSphere DataStage and InfoSphere QualityStage.

Note: Same way ,you can assign riles to groups also.It is better to assign roles to groups instead of assigning to individual users.

If you want to add any other user as administrator who is not part of any group,Then you need to assign administrator role to that user individuals.

How to export/import Projects in datastage

                  How to export/import Projects in DataStage

There are multiple ways to export/Import projects in DataStage. We will discuss here two ways.

1. To use the istool command line interface (CLI) to export projects
2. To use InfoSphere DataStage Designer client to export projects

1 istool---->   You can use istool to export and import projects in datastage. Benefit of using istool is that you can export/import the whole project in one time.You can also create a script and schedule it.

You don't need to manually export each project.You can specify the location of your projects and put it into a loop.

istool.sh is inside /opt/IBM/InformationServer/Clients/istools/cli/

Here are some export commands.

To Export Projects/Jobs:

If you want to export some specific job which is in folder "Jobs/abc Jobs/".Then the command would be

istool export -dom SERVER:9080 -u admin -p admin -ar /test1.isx -ds ' "SERVER/Test/Jobs/abc Jobs/jobname.pjb" '

Server name  is where you datastage is installed.9080 is the dault port but it may be different.If you are using default port,you dont need to specify the Port number in command.

If you just want to see the number/list of jobs exported,then you can use preview command.It will only display you the list of jobs exported without exporting them.

istool export -dom SERVER:portnumber -u admin -p password -ar /test1.isx -ds ' "SERVER/Test/*/*.*" ' -preview

If you want to export whole project

/opt/IBM/InformationServer/Clients/istools/cli/istool export -dom servername:portnumber -u userid -p password -ar "Project_name.isx" -ds "servername/Project_name/*/*.*" >> "Project_name.log"

File created with istool will have .isx extension.

Syntax and additional options can be done by executing istool -help. An example to get export help/syntax is below.

istool export -help
istool export [ Options ]

To Import Project

To import whole project

 /opt/IBM/InformationServer/Clients/istools/cli/istool.sh import -dom servername:Port number -u User id -p password -ar "project_name.isx" -replace -ds  "servername/Project name"  >> "Project_name.log"


If you want to import test1.isx file into a project called Test.command would be

istool import -dom SERVER:portnumber -u admin -p password -ar /test1.isx -ds ' "SERVER/Test" '

2. Datastage designer

To Export Project

You have to open the data stage designer with the project which you want to export.

1. Login into desktop designer with the Project from where you want to export.
2.  select Export item from  menu.
3. This brings up the Repository Export dialog. All items in the project are automatically selected in this dialog.
4. Click on Add to add the jobs to export.
5.  Specify the  path of the export file name.For example,D:\Temp\project_name.dsx.
6. Choose export without executable.
7. This file is created by the export process. For example,D:\Temp\project_name.dsx.
8. Select Export.
9. Select Close when the export is complete.
10. File created with data stage designer would be with .dsx extension.

To export other projects, close the InfoSphere DataStage Designer client and open theInfoSphere DataStage Designer client against another project to repeat the export process.


To Import Project

1. Move the .dsx files that you created in previous steps to the computer where the InfoSphere DataStage clients are installed(if the export was done on some other machine).
2. Open the InfoSphere DataStage Designer client and attach to the target project.
3. Select Import > DataStage Components.
4. Specify the name of the .dsx file to import, and click OK.
5. Import.

DataStage

                   

                                   DataStage Installation

Before starting Data Stage installation,You need to run ISALite so that you can verify that your server fulfill all the requirements for datastage installation.

1. Download ISALite as per your OS and run ISALite.exe .
2. Run ISAlite and save the output in a file.
3. Read the output file to see if there is any failure.If there is any failure in the output report then resolve it before installing datastage.

Prerequisites       

1. It is recommended that you should install datastage in /opt/IBM/InformationServer.
2. InformationServer should be empty before installation.Otherwise Installation will give error.
3. You should have all these accounts on the server before starting installation.These accounts should be local on the server.

Userid	Group	Description
dsadm	dstage	This account will be Administrator for  the InfoSphere Information Server engine
xmeta	dstage	Metadata repository owner
xmetasr	dstage	Staging area user
iauser	db2sadm1 (Local group)	Owner of the Information Analyzer analysis database
srduser	dstage	Owner of the QualityStage Standardization Rules Designer database
db2inst1	db2iadm1 (Local group)	DB2 instance owner

db2fenc1	db2fadm1 (Local group)	DB2 fenced user
dsodb	dstage	Owner of the operations database

  Other than these users you need two more users isadmin and wasadmin.These accounts can be LDAP/AD accounts.

4. You need below privileges to install Datastage on different OS.

Operating system                               Accounts

AIX®                                                The root user account.
Linux

Solaris

Windows                      
A user account in the local Administrators group. This account must be one of the following types:
A local account that is directly assigned to the Windows local Administrators group.                             A domain account that is directly assigned to the Windows local Administrators group.

5. You have to decide before installation that if you want to use DB2 as repository or you want to use some other DB.You have to choose this option during installation.

6. Before installation,You need to install all the required clients for all the database which you will use as data source or repository.

Steps for DataStage Installation

1. In the newly created session, run the following command to clear any Java™ options that might be      set.

      Linux

         (i) In the sh or bash shell:

         export IBM_JAVA_OPTIONS=

        (ii) In the csh or tcsh shell:

         unsetenv IBM_JAVA_OPTIONS


2. AIX Solaris Linux: Configure the following settings for the session to help ensure that the                    installation is successful:

         (i) Set the file creation mask to 022:

                   umask 022

            You can just type umask,It will show you what umask properties are set.

       (ii) Ensure that you have no less than 10240 file descriptor resources allocated.

                 You can type ulimit -n to see what properties are set

      (iii) Check that the current setting for the NOFILES kernel parameter is set to a value that is at         least equal to the value that you specify for ulimit.



3. Extract the installation package. For example
     
       Linux
       tar zxvf is-suite.*.tar.gz



The package is named is-suite.* for the full product or is-client.* for the client only.

4.  The is-suite images require the use of a Bundle Entitlement package; the is-client images do not. If you are installing an is-suite image, extract the contents of the entitlement package into the is-suite directory that is decompressed in the previous step. Ensure that you have downloaded the correct entitlement package for your operating system platform, licensed products, and suite installation package. A suite entitlement package will only work for a suite installation package. For example:
unzip -d is-suite Bundle.suite.*.zip


5. Change to the is-suite or is-client directory.

6. Run the installation program:

Operating system Command

AIX Solaris ./setup [-verbose]
Linux ./setup [-verbose]

7. The -verbose option is optional. The option causes the installation program to display log messages to the console and also write them to the log file. If the -verbose option is not specified, log messages are written to the log file only.
Several messages are displayed. The program then displays a message about how to access the installation wizard:
======> Enter the following URL to your web browser to begin the
installation process:
https://localhost:8443/ISInstall

8.  Open a web browser on the computer or on a remote computer, and navigate to the URL that is listed in the message from starting the installation program. (Use the full host name if using a remote browser.)

9.

10. A message might appear in the browser that reads "The publisher cannot be verified..." You can safely ignore the message and continue the installation. This message does not indicate a security risk when it appears during InfoSphere® Information Server installation.

• Select a language from the menu for the installation program to use, and click Login.
• This selection applies only to this installation session. Later in the installation process, you select the language to apply to the InfoSphere Information Server installation.
• Follow the prompts in the installation program to continue installing the product.
• The installation program runs an early prerequisites check to ensure that your installation is configured correctly. 
• The wizard page displays each check. If the check fails, it is marked FAILED in the page. If the check found issues that might cause problems with the installation, it is marked WARNING in the page.

11. When all prerequisites checks pass, continue to follow the prompts in the installation program.

12. You can follow the progress of the IBM® InfoSphere® Information Server suite installation by monitoring the installation log files.
     
       Open the log file:
   
       Use the tail command       tail -f ISInstall*.log

        ISInstall*.log is the log file with the most recent date timestamp.

   Log file should exist at /opt/IBM/InformationServer/logs.

   You can monitor the installation of the following InfoSphere DataStage  components:

• InfoSphere DataStage plug-ins
• Parallel engine operators
• Connectors

Open the log file at /opt/IBM/InformationServer/Server/StagingArea/Log



After Installation Steps


1. Installing the new version of InfoSphere Information Server on the client computer

2.Use IBM® Support Assistant Lite for InfoSphere® Information Server (ISA Lite for InfoSphere Information Server) to run diagnostic tests and generate reports that confirm the stability of the installation environment.

Generate the Diagnostic Health Checker reports

Note: If you already have ISALite installed, verify that it's the latest version. To verify the version that you have installed:
• On Linux and Unix computers, issue the ./runISALite.sh -version command from the install_dir/ISALite directory.

3. Check if components are running after installation.

datastage

      Cd  /opt/IBM/InformationServer/Server/DSEngine
       ps -eaf | grep dsrpc

websphere
       ps -eaf | egrep -i "java"

ASB Node
       ps -ef | grep -i agent

(ii). Check version.xml file for success.

    Go to Cd  /opt/IBM/InformationServer

    Cat version.xml

(iii). Check the version installed
   
   Cd /opt/IBM/InformationServer/Server/DSEngine
   
      cat .dsrel

       8.7.0.0

4. Saving InfoSphere DataStage settings files

Procedure

(i). Complete these steps to save the required settings files: If you are installing datastage on a different server,You don't need to copy these files to some other location.You can copy the contents of these files from old server to new later.

    a. Copy the $DSHOME/dsenv file to a location that is outside of the InfoSphere DataStage file structure.(if you are installing new version on the same server)

    b. To keep a record of the existing configuration and the configuration of the ODBC drivers, copy the following files to a location that is outside of the InfoSphere DataStage file structure:

•          $DSHOME/.odbc.ini
•           $DSHOME/uvodbc.config
•           The uvodbc.config file that is located in each project directory

(ii). Complete these steps to save the required settings files: 

       a. Copy the DSParams file from each project directory to a safe location.

       b. Copy the DSParams file from the Template project. For example, save the                                                C:\Ascential\DataStage\Template\DSParams file.

       c. Copy the $DSHOME\uvodbc.config file in the engine directory.

       d. Copy the uvodbc.config file that is located in each project directory.


5. Check the basic connectivity with the databases to make sure everything is working fine.

To test basic connectivity with Oracle-

export Oracle_home variable
export Path variable.
sqlplus username@databasename
password

If you are able to connect then basic connectivity to oracle is working fine.

To Test Basic Connectivity with teradata--

You can use any teradata utility like bteq,fastload etc

bteq
.logon databasename/username
password

If you are able to connect then connectivity to teradata is working fine.If not then you may need to check if teradata client is installed properly on the server or not.

6.  Now export/Import the projects

7. Make your dsadm and repository password no expiry if possible.This will reduce overhead of changing and updating passwords.