Data Stage Jobs are hanging after Upgrade/Installation

           

               Data stage jobs are hanging on new server (after upgrade)

 If you are facing below error-

After Data Stage upgrade or new installation, if your jobs are hanging during testing, hundreds of jobs run together but don’t get executed and get hanged. Jobs are taking a lot of time in execution.

This problem may be because the nproc value and max user processes variables don’t have enough values to execute the jobs.

[root@localhost# ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 127358
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65535
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

A decent value for max user processes should be 10K.Please check above values for dsadm account or the account which is running your Data stage server.

If open files and max user processes values are less then and get it fixed by Linux team.

Nproc values are saved in /etc/security/limits.conf .

If you are upgrading on a new server,You can verify these settings on your current server also.

How to export/import Projects in datastage

                  How to export/import Projects in DataStage

There are multiple ways to export/Import projects in DataStage. We will discuss here two ways.

1. To use the istool command line interface (CLI) to export projects
2. To use InfoSphere DataStage Designer client to export projects

1 istool---->   You can use istool to export and import projects in datastage. Benefit of using istool is that you can export/import the whole project in one time.You can also create a script and schedule it.

You don't need to manually export each project.You can specify the location of your projects and put it into a loop.

istool.sh is inside /opt/IBM/InformationServer/Clients/istools/cli/

Here are some export commands.

To Export Projects/Jobs:

If you want to export some specific job which is in folder "Jobs/abc Jobs/".Then the command would be

istool export -dom SERVER:9080 -u admin -p admin -ar /test1.isx -ds ' "SERVER/Test/Jobs/abc Jobs/jobname.pjb" '

Server name  is where you datastage is installed.9080 is the dault port but it may be different.If you are using default port,you dont need to specify the Port number in command.

If you just want to see the number/list of jobs exported,then you can use preview command.It will only display you the list of jobs exported without exporting them.

istool export -dom SERVER:portnumber -u admin -p password -ar /test1.isx -ds ' "SERVER/Test/*/*.*" ' -preview

If you want to export whole project

/opt/IBM/InformationServer/Clients/istools/cli/istool export -dom servername:portnumber -u userid -p password -ar "Project_name.isx" -ds "servername/Project_name/*/*.*" >> "Project_name.log"

File created with istool will have .isx extension.

Syntax and additional options can be done by executing istool -help. An example to get export help/syntax is below.

istool export -help
istool export [ Options ]

To Import Project

To import whole project

 /opt/IBM/InformationServer/Clients/istools/cli/istool.sh import -dom servername:Port number -u User id -p password -ar "project_name.isx" -replace -ds  "servername/Project name"  >> "Project_name.log"


If you want to import test1.isx file into a project called Test.command would be

istool import -dom SERVER:portnumber -u admin -p password -ar /test1.isx -ds ' "SERVER/Test" '

2. Datastage designer

To Export Project

You have to open the data stage designer with the project which you want to export.

1. Login into desktop designer with the Project from where you want to export.
2.  select Export item from  menu.
3. This brings up the Repository Export dialog. All items in the project are automatically selected in this dialog.
4. Click on Add to add the jobs to export.
5.  Specify the  path of the export file name.For example,D:\Temp\project_name.dsx.
6. Choose export without executable.
7. This file is created by the export process. For example,D:\Temp\project_name.dsx.
8. Select Export.
9. Select Close when the export is complete.
10. File created with data stage designer would be with .dsx extension.

To export other projects, close the InfoSphere DataStage Designer client and open theInfoSphere DataStage Designer client against another project to repeat the export process.


To Import Project

1. Move the .dsx files that you created in previous steps to the computer where the InfoSphere DataStage clients are installed(if the export was done on some other machine).
2. Open the InfoSphere DataStage Designer client and attach to the target project.
3. Select Import > DataStage Components.
4. Specify the name of the .dsx file to import, and click OK.
5. Import.