Upload

Guidelines for Programmers

Updated 7/22/2013

  1. Creating an XML data file requires knowledge of the database structure that the data is stored in, and requires basic to intermediate computer programming skills.

  2. Knowing how to read and validate against an xsd is a key skill for successfully completing an XML upload. You can find additional information on how to read and use an XSD at The XML Specification (W3C), An XML tutorial (w3schools) and An XSD tutorial (w3schools)

  3. For the new schema types (CTv3, aggregate and client level), the entities (agencies, sites, interventions, and programs) must already exist in EvaluationWeb. To create/manage these entities, use the AgencyInfo XML file, or use the Manage Agencies button in the user interface (available to administrative users in the menu on the left).

  4. EvaluationWeb uses various IDs for a number of different purposes. First, they're used to uniquely identify a test event, client, site, intervention, agency, etc. Second, they're used to tie a record in an uploaded file to an agency, site, intervention, worker, or existing record. Generally, the term Local ID is used in the system for the value we will be attempting to match. For example, in the EvaluationWeb Agency Setup window in Manage Agencies, there is a field labeled “Local ID.” If you will never be uploading files for a particular agency, this field may be left blank (if you will never upload any files) or you may enter "0" in the field (indicating that records from this agency will not be in any files you do upload).

    If you will be uploading data for an agency, the Local ID in the EvaluationWeb Agency Setup window must match the AgencyID in the files you upload. The same is true for other ID types (sites, interventions, etc.). The ID can be whatever you want (an old PEMS ID, an internal numbering, etc.), but it should be consistent with IDs in files sent to CDC before EvaluationWeb was in use. For new agencies, sites, etc., the only ID requirements are that the EvaluationWeb database must reflect the ID you choose, and the ID cannot duplicate an existing ID for that type of record. For example, if there is an agency in the database with the AgencyID "123", you cannot add a new agency also having the Agency ID "123." (But note that Site and Intervention IDs only need to be unique within the agency they are in.) There are several terms that we may use to refer to an ID: localid, XXXXXid or localXXXXXid can refer to the same thing. For example, localid, clientid, and localclientid can all refer to the same thing.

  5. Aggregate level, Client level, and Partner Services are generally very strictly enforced (almost all rules trigger an error).

  6. Each format has a core record level. (For example, Partner Services uses "client"; HIV Testing uses "HivForm"; the Aggregate data formats use "aggregateData"; and, the Client Level formats use "client".) Each of these types has unique identifying information (formId (CT), clientId (CL), localClientId (PS), or period and organization for Aggregate). These are generally considered unique at the agency level (two test records with the same formId from the same agency are considered to be the same event, while the same formId from different agencies is not). In order to be able to update records, the unique identifier in the uploaded file must exactly match a record already in the system. For example, to update information on client 123 in agency ABC using v2:

    <agency agencyId="ABC">
    	...
    	<client status="U" localClientId="123">
    	... Put actual updated record here ...
    	</client>
    	...
    </agency>
  7. Note that the full record must be included every time it's uploaded. This is because it's generally not possible to tell an update to the lower level entries from a new entry. For example, in order to add another session to a client's record, all sessions for that client must be listed in the updated record. Since there are no ids associated with a session, we treat the session date as the identifier. A session with a date we haven't seen before could be an update to an existing session or a new session. And there's no way to tell which session is being updated in the update case.

  8. If a section lists optional data that is often not collected, it might not be included in the sample files. (For example, worker information is not included in the Partner Services v1 sample file.)

  9. Since uploads match existing entities based on ids, we require that all entities in the system have an id assigned (in order to prevent duplication). This is announced upon entering the upload screen, and missing ids will prevent uploads. Ids can be added through Manage Agencies in menu on the left.