Plans for Farms and SAM

Hardware available

The nodes are:

• fcdf171-200 The 30 nodes we are testing. to be increased to 80
• fcdf171 The head node
• fcdfdata057: Disk server with SAM cache. Has the sam station cdf-farm1 running on it (stager, fss, station, gridftp) and a pnfs mount. More details are listed below.

These 80 nodes are sufficient for the streams with which we wish to run SAM.

There are 80 more nodes available that we would like to set up with JIM as the submission system. The batch system can be chosen at will -- JIM adapts and idealizes the batch system. Obviously it would be useful to pick the same one as D0: fbs. That is also the one that Steve Timm's group has the most experience using. A decision to move to Condor can be taken later if the division so decides and we do have adaptors for that.

Interaction of farms and SAM

The Farms interaction with SAM is limited to:

      get next file
      sam cp
      release file

Scalability

The idea is to keep the farms at 80 to 100 nodes and federate with JIM. So that should be scalable. Farms process 1 file every 3 hrs so 100/180 is 1.8 minutes per file request. Each file is 1G. So that seems to be placing a light demand on getting files. Files can be obtained from tape or via a dcache door (as configured by sam_cp). If desired, separate identical stations can be configured to prefer one or the other, depending on whether one is obtaining data from tape or cache.

Interaction with analysis CAF

Several considerations here, they become less and less concrete as we go from 1 to 4.
• 1. We want to access raw data
• 2. We want to access data in golden pools for reprocessing since we can reprocess from our processed data
• 3. We will write our calibration streams into DCache and Tape at the same time so we can avoid a tape access if we get them from DCache
• 4. We eventually want to use dcache write pools and avoid any access to tape. This saves on tape drives needed.
Implications
• 1. This is mostly not in dcache so we access it via pnfs mounts on a standard sam cache.
• 2. We can access this via a dcache gridftp door, just as we do for offsite. Is one door enough? From the computation above, if it takes 3hr/job, probably ok
• 3. This could also go to the door or we can configure to access dcache-as-sam-cache with another station on fcdfdata057. (we specify the station when we start the job) Unfortunately, this means that we never get the dcache for cdf synchronized since 2 stations are claiming it as cache (yuck). Interesting grid problem.
• 4. Same comment as 4.

The migration to JIM

Valeria has had success in installing JIM in Oxford for MC and the production usage of JIM is working for D0. Gabriele is available after Jan 15, when he returns from vacation, to work on JIM issues for CDF. A major issue has been configuration and Alan Sill is available to work on packaging and configuration.

JIM is the right step in the direction of FermiGrid. The configuration with sam caches and servers on each node is not required for JIM and the step to the JIM-based Farm for CDF can include an intermediate marriage with SAM stations or not. This depends on the state of the art. Currently without a SAM Cache, JIM uses GridFTP to copy the file; however, robustness requires

• retries with exponential backoff
• CRC checking
These are built into sam_cp but configuration issues need to be clarified by Alan before we can use this. We also need to discuss this point with Gabriele.

Thus these nodes can be brought up with JIM and when that is working, we can migrate the CAF-Condor nodes to JIM. Then these two JIM installations can be federated through brokering that is the development task to which Valeria is assigned.

Demands on the SAMGrid Team

All of this will be based on SAM V5. It is now clear that the memory leak plugging will take time for SAM V6 and it is not going to be in a production state. In addition, JIM has not been modified to work with SAM V6.

From the SAMGrid point of view, V5 production support is fine, unless V5 production support is construed to mean wide open feature changes. We need to assess the need for feature changes expected. However, SAMGrid has to

• establish v5 integration testing to allow bug fixes to be tested before deployment.
• set up v5 product suites that are testable in the integration.

For the second case, we have checked that the current V5 software as released in sam_products v2_15 is the correct set for station installation. Alan is creating a version of sam_client_products that conforms to Lauri's collection of the latest client (cdfsoft) side. To use this with cdfsoft as it stands, it is required to explicitly setup sam v5 -q prd-old. This will be available Monday, December 13, but these products can be easily installed on fcdfcode1 by hand.

Farm Configuration Details

The configuration is determined using the SAM Station Configuration Instructions .

Constraints:

• Station cdf-farm1 on fcdfdata057.
• SAM account will be exported to all worker nodes, fcdf171-200.
• 10G of sam cache on each node
• Must be able to get files and store them from the workers.
• There is a large cache (2TB) on fcdfdata057.
• fcdfdata057 is mounted to pnfs.
This leads to the following configuration for the station node:

[sam@fcdfdata057 sam]$ more private/fcdfdata057_server_list.txt
station prd v4_2_1_72 cdf-farm1 --preferred-loc=/pnfs/cdfen/filesets --min-delivery=1k --constrain-delivery=fcdfdata057.fnal.gov
stager prd v4_2_1_72 cdf-farm1
fss prd v6_0_1_7  cdf-farm1
gridftp prd v2_0_12

And for the worker nodes:

stager prd v4_2_1_72 cdf-farm1 --with-sm --without-fss --log-file=stager_log --max-transfers=5

Software version on Station node

 bash-2.05a$ more products/sam_products/v2_15/NULL/ups/sam_products.table
  File       = Table
  Product    = ANY
  Flavor     = ANY
  Qualifiers = " "

  ACTION = SETUP
 
  proddir()
  setupenv()
 
  setupRequired("-j perl                v5_006_1a   ")
  setupRequired("-j globus_location     v2.0        ")
  setupRequired("-j globus_dh_server    v2_0a       ")
  setupRequired("-j globus_dh_client    v2_0a       ")
  setupRequired("-j sam_gridftp         v2_0_12     ")
  setupRequired("-j sam_gsi_config      v2_1_2     ")
  setupRequired("-j sam_gsi_config_util v2_0        ")
 
  setupRequired("-j sam_config        v4_2_32       ")
 
  setupRequired("-j sam               v5_0          ")
  setupRequired("-j sam_batch_adapter v0_9_9_5      ")
  setupRequired("-j sam_bootstrap     v4_2_28       ")
  setupRequired("-j sam_config_tools  v0_1          ")
  setupRequired("-j sam_cp            v4_0_25       ")
  setupRequired("-j sam_cp_config     v4_0_8        ")
  setupRequired("-j sam_common        v5_1_1_4      ")
  setupRequired("-j sam_dcache_cp     v5_2          ")
  setupRequired("-j sam_kerberos_rcp  v4_0_11       ")
  setupRequired("-j sam_station       v4_2_1_72 -q GCC-2.95.2")
  setupRequired("-j sam_user_api      v5_1_0_5      ")
 
  setupRequired("-j fnorb   v1_1b_8              ")
  setupRequired("-j orbacus v3_3_4 -q GCC-2.95.2 ")
  setupRequired("-j python  v2_1                 ")
  setupRequired("-j setpath v1_11                ")
  setupRequired("-j tcl     v8_3_1               ")

Client Software

The software reflects that in production d0db-prd:

sam v5_0 -f NULL -z /fnal/ups/db
|__sam_user_api v5_1_0_4 -f NULL -z /fnal/ups/db -g current
    |__python v2_1 -f SunOS+5 -z /fnal/ups/db -g current
    |  |__blt v2_4u -f SunOS+5 -z /fnal/ups/db
    |     |__tk v8_3_1 -f SunOS+5 -z /fnal/ups/db
    |        |__tcl v8_3_1 -f SunOS+5 -z /fnal/ups/db
    |__fnorb v1_1b_11 -f SunOS+5.6 -z /fnal/ups/db -g current
    |__sam_common v5_1_1_4 -f NULL -z /fnal/ups/db -g current
    |__sam_batch_adapter v0_9_9 -f NULL -z /fnal/ups/db -g current
    |__sam_config v4_2_28 -f NULL -z /fnal/ups/db -g current

Also, the sam_admin on d0db-prd is v5_1_0_6.