Notes from the April 8 Workshop Greg's Notes on Progress Slides CDF has June deadline for running with Jim. Currently, Jim uses older version (non-ShahKar) of RunJob, but Dave's initial CDF work is based on ShahKar. ==> The June 15 deadline for DZero switchover to ShahKar should -Run Reprocessing -MC Production with Jim -Ordinary user ==> This timing matches the CDF plans to run on JIM in June. -Also needs CDF apps working, building on what Dave had done already for CDF. --> We should follow up with meetings with SAMGrid team on the way to this milestone. (One before the end of April.) Greg's notes on feature requests not covered below: The flexibilty of defining metadata within RunJob is undesirable from the standpoint of folks who need to know exactly what is being run. So we introduce locking of Configurators. There is already a simple interface for doing this. Keeping track of what they actually used: there is an interface for doing this that needs to be brought up to date. (Essentially the Configurator Print() method.) Validation API: This is a very interesting new idea! RunJob should communicate what it is going to do back to the request system just before the job is submitted and optionally check with it afterwards. (DZero and maybe CDF store input parameter data dirctly in the files. CMS does this in some cases.) Requires cooperation with other tools. CMS does some of this already at the end of jobs by sending summary files back - but it is currently "outside" of RunJob. Validation in the workflow: accommodation of a breakpoint between production and validation steps to do validation. Where do the parameters come from? We have to be careful that people know what is being specified! More feedback from the context mechanism is important at the user level. Make this specific to certain context files/configurators so that users can get back useful information about exactly how things were configured without it being a firehose. Notes on Interfaces (mainly by K. Wyatt Merrit with editing by Greg) Runjob Requirements and Interfaces from different customers. Round 1. The goal is to have an agreed-upon set of requirements by 6/15/04. We will have a functioning product before then, so these requirements will signal a new round of development. Development currently going on will try to anticipate these requirements. This is a proposed outline for a document to meet the requirements milestone. I. List of customers and use cases by customer. II. Description of basic role of Workflow Management layer from each customer. III. Specific Workflow Management requirements from each customer by use case IV. Suggested reconciliations of different requirements, if any Heres a rough draft of what we can fill in of the outline from Round 1. I. Customers and Use Cases Customer Use Case Run II Data Handling Production jobs submitted with SAMGrid Analysis jobs submitted with SAMGrid Analysis jobs submitted to local resources Production jobs submitted to local resources (hope this use case is interim) CMS Data Handling Production jobs submitted with LCG Tools Analysis jobs submitted with LCG Tools There will be an interim period where support for LCG Tools will also include support for Hybrid tools in the US. (We're in this interim period now.) II. Basic role of Workflow Management The basic role of workflow management in Run II Data Handling is to provide mechanisms to conduct a sequence of jobs, chained by the output of one becoming the input of the next job in the sequence, using one or more experiment-defined mechanisms for job description, and one or more experiment-defined mechanisms for providing job RTEs. The role in CMS is similar. III. Specific Requirements from Customers The interfaces needed by Run II Data Handling for SAMGrid use cases are: 1. Workflow Management should be able to accept a SAMGrid Request ID as the source of the job definitions which determine the specific input parameters and executables to use in the workflow sequence. There are some associated requirements having to do with verification of input parameters, possible ways to override or to disable overriding of parameters, and storage of actual parameters used. These requirements should be considered tentative at this point; discussions with the Run II experiments are ongoing. 1.a. Workflow Management should be able to return to the Request System the actual input parameters used (or a reference to a storage location for those parameters? the cardfiles are what gets overridden. The Request System does not store the cardfiles but a reference to a cardfile in cvs is that right?) 1.b. Workfile Management should have an interface to disable any parameter override mechanisms. This is required specifically for the production use case, so that the production requestor has confidence in the ability to completely specify the job without ambiguities. 2. Workflow Management should be able to accept one or more SAMGrid datasets as input data or subsidiary input data (e.g., minbias files to overlay, or calibration files too large for the RTE tarball) to any of the jobs in a job sequence. 3. Workflow Management should be able to store its final and/or intermediate output files using a SAMGrid command. This implies being able to generate SAM catalog compatible metadata for ALL the intermediate files (in order to have a complete parentage record), even if not all intermediate files are actually stored with a SAMGrid location. 4. Workflow Management should be able to accept a SAMGrid dataset as the source of the run-time environment tarball(s) for its jobs. 5. Workflow Management should be able to have a breakpoint between production and validation steps, and should be able to condition data storage from production based on validation results. [Further requirements from use case of analysis on local resources, TBD] [Further requirements from the possibility of d0tools using some Runjob components, TBD or should it be vice versa?] The interfaces needed by CMS Data Handling are: (This needs discussion within CMS.) Runjob notes. Dave Evans. Cut off support for V6 (?). Writing code for V7. Basic prototype working by end of month. Questions: deployment plan for V7 for all use cases: reco reproc, MC proc, individual user, CDF proc Question: details of changes communicated back to SAMGrid soon.