Computational Nodes

The computational (“slave”) nodes are based on the SuperMicro DLE motherboard in a 2U rack mounted case.  The systems have two 800MHz Intel Coppermine PIII CPU’s and 512MB of 133MHz SDRAM.  The Coppermine CPUs have full speed cache, instead of the half speed cache on previous Pentium III versions.  Benchmark tests, on evaluation systems using CMS reconstruction code, revealed approximately 5% improvement in performance using Coppermine CPU’s with 133MHz memory, over previous Pentium III versions using 100MHz memory.  The choice of motherboard was based on the availability of a system which could support dual Coppermine CPU’s and 133 MHz SDRAM, while not including a SCSI controller (which is an unnecessary and costly for the computational nodes).  The CPU speed was chosen to be at the best price performance point.  The amount of memory was chosen to be adequate for two simultaneous ORCA production jobs, with a small amount in reserve for future increases in the ORCA image size. The computational node motherboards can address up to 4GB of RAM, but 512MB is seen as sufficient for now.  A number of vendors were asked to provide cost estimates for assembling the nodes: in the end the two lowest cost vendors offered the same price. Datel Systems of San Diego was chosen because they offered a better service arrangement.

Server Nodes

Caltech and UCSD use different servers to attach the RAID arrays. Caltech uses twoservers. The primary server is a Dell PowerEdge 4400, a similar model to which we have been using extensively in other projects, and found to be reliable with excellent I/O and CPU performance. The secondary server is a dual processor Pentium III 933 MHzsystem with 1 GByte memory, and integrated dual 3ware Escalade 6800 ATA RAID controllers.This server also contains dual SysKonnect SK9843 single port Gbit ethernet cards, as doesthe Dell PowerEdge. The secondary server contains about 1.2 TByte of ATA disk in the formof 16 IBM DeskStar units each of 75GB. These disks are configured in RAID0 (see next section).UCSD uses a 4U custom configured rack mounted system based on the SuperMicro DE6 motherboard, which has a much lower cost than the Dell.  Both are dual CPU systems with 2GB of SDRAM.

The following Figure shows the UCSD Tier2 equipment.

Disk Arrays and Tests

Substantial quantities of disk space are required at each of the Tier2 prototype centers. An initial capacity of 2 TBytes at each site was foreseen. We have chosen SCSI RAID as the best technology from price vs. performance as well as availability and reliability considerations. We evaluated several different configurations before choosing a vendor:

• JBOD Seagate and IBM disks with software RAID
• FibreChannel Seagate disk arrays from nStor Corp. with hardware RAID
• Dell PERC hardware RAID with IBM and Seagate disks
• Adaptec I2O hardware RAID with IBM and Seagate disks
• Software RAID with IBM and Seagate disks
• Hardware RAID from Winchester Systems
• Software RAID (1) with Hardware RAID (5) from Winchester Systems
• Hardware RAID (0,1,10,5) from 3ware using ATA disks

The evaluation tests involved measuring the capability of the RAID array to sustain high I/O rates with a variety of block sizes, and the real throughput from an Objectivity database application when writing and reading to the RAID array. The best performance was achieved using the Winchester Systems FlashDisk.

Winchester “FlashDisk”

Showing the two Winchester “FalshDisk” arrays, each of 500GB.

The Winchester “FlashDisk” RAID array we tested is a ruggedized rack-mountable disk enclosure of height 4U, containing 8 SCSI 75Gbyte Ultra160 Seagate disks and a RAID controller. The controller was initially an Ultra II (80 Mbytes/sec), but was replaced with an Ultra160 (160 Mbytes/sec) version, which only recently become available in the industry. We configured the 8 disks as two RAID5 arrays of 4 disks each. Each of the arrays was connected to a host Dell PowerEdge 4400 server on one of two dedicated Adaptec 39160 Ultra160 channels. The two arrays were then striped at RAID1 in software, to arrive at a virtual disk of approximately 0.5 Tbyte in capacity. This virtual device was capable of very stable read and write speeds of a consistent 116 MBytes/second using a memory-disk-memory performance measurement tool[2] and approximately 35 MBytes/sec read and write speeds using our Objectivity/DB-based test application[3] running a single thread. In a third test, using Java I/O in three parallel streams, we achieved 70 MBytes/second by simultaneously reading three database files of 1Gbyte each. The results with the Winchester FlashDisk were much better than any of the other configurations listed above. An additional advantage of the FlashDisk is that we are free to ourselves insert commodity SCSI disks of larger capacity as and when they become available. Furthermore, no software drivers are required for the FlashDisk, as the RAID controller emulates a simple SCSI interface, which means that the array appears as

a standard SCSI device to a host OS.

Winchester FlashDisk Test details

Windows 2000 Server

–         Memspeed_v2 (Jim Gray)

•          Single Ultra2 Array

–         Read @ 67 MB/sec

–         Write @ 66 MB/sec

•          Single Ultra3 array

–         Read @ 97 MB/sec

–         Write @ 89 MB/sec

•          Software NTFS Stripe two Ultra2 Arrays

–         Read @ 116 MB/sec

–         Write @ 115 MB/sec

–         Java IOSpeed (JJB)

•          Read one 5GB file @ 53 MB/sec [10 MB buffers, Ultra3]

•          Read two 5GB files @ 74 MB/sec

Linux 2.4 with new SCSI drivers

–         Dsktst6 (Jan Lindheim)

•          Ultra3

–         Write @ 70 MB/sec

–         Java IOSpeed (JJB)

•          Read one 2GB file @ 33 MB/sec [reiserfs, 1MB buffers]

ASA Server – 3ware “Escalade” 1.2 TByte Array

ASA Computers built 22-disk bay, dual 933MHz PIII, 1 GB memory, one SCSI system disk, twin SysKonnect SK9843 Gbit cards

Two 3ware Escalade 6800 PCI controllers, each with 8 IBM 75 GB ATA100 disks: 1.2 TB usable capacity

Disks on each 6800 can be configured in RAID0, 1, 10 or 5. All RAID is done in hardware.

Custom built - very heavy, well cooled

Pre-installed with RedHat Linux 6.2

We paid 14k$ for all the above

ASA Server Test Details

•          Hardware RAID0

–         4 disk set, write @ 52 MB/sec

–         8 disk set, write @ 65 MB/sec

•          Software stripe of two Hardware RAID0 sets

–         Two sets of 4 disks, write @ 68 MB/sec

–         Two sets of 8 disks, write @ 73 MB/sec

•          Hardware RAID5

–         4 disk set, write @ 2.8 MB/sec

–         8 disk set, write @ 6.0 MB/sec

•          Consistent with 3ware results opposite

nStor FibreChannel 1TB array

•         One tray of eighteen 18GB disks

•         One tray of eighteen 36GB disks

•         One controller

•         Total of 1 TB

•         RAID5 and RAID0 being used

•         Hosted on Sun Enterprise 250 using LightSpeed LP8000 PCI-FibreChannel adapter

•         Good controller, good disk cooling.

•         Poor disk mounting system

•         Best read rate seen: 30 MB/sec

•         Best write rate seen: 7 MB/sec

•         Poor performance blamed on LP8000 Solaris driver

•         But similar results obtained under Win2000 server using the Windows driver

•         Smoking!

Other Disk Arrays

•         We tested an Adaptec “I3O” PCI RAID controller with four Seagate Cheetah 15,000 rpm Ultra3 disks. Using RAID0 the best we saw was read @ 23 MB/sec and write @ 14 MB/sec i.e. worse than using a single disk!

•         In another test, using Windows 2000 software striping, and an array of Seagate disks, we saw read @ 55 MB/sec and write @ 35 MB/sec using default settings and without further experimentation.

Network Configuration

Both Caltech and UCSD are preparing to use low cost, medium performance network switches from HP, which will be connected via the servers to existing high performance switches at CACR and SDSC.  The HPSS and external network at both facilities are connected to Gbit Ethernet switches.  To network the Tier2 prototype components, the computational nodes are connected to an HP ProCurve 2524Ethernet switch (auto sensing 10/100 Mbit on each of 24 ports) equipped with two Gbit ports. One or both of the Gbit ports will be connected to the server, which in turn (and on a separate circuit) is connected to the existing high performance Gbit Ethernet switches at SDSC and CACR. 

Leverage Resources:

The Tier2 prototype project is leveraging resources (both in terms of infrastructure and people) from CACR and SDSC.  In terms of facilities the project benefits from high speed networking, both locally at each site and between the two sites; an HPSS at each site; and environmental services like cooling and power conditioning.   The project also benefits from considerable expertise in the staff at CACR and SDSC in the areas of assembling, configuring, and managing Linux clusters.

Hardware Activities

January 2001-March 2001

• Node cloning completed
• ATA Disk array installation, configuration and testing
• Winchester Disk array installation, configuration and testing
• Gbit connections to CACR HPSS and High Performance Network from Tier2 (main server) and Tier2b (secondary server, ATA disk array server) completed.

Software Activities

January 2001-March 2001

• CMS Software Configuration, Installation, and Maintenance – ORCA versions 4.3.0.pre4, 4.3.2, 4.4.1 installed, IGUANA 2.1.0, 2.1.2, 2.3.0 installed
• System Configuration for CMS Production
• Lots of work from Vladimir automating CMSIM Production and ORCA Production
• Installation of GDMP 1.2.1.
• GDMP Stability Testing – replication of 170GB of Minimum Bias events from CERN to Caltech Tier2.
• Production of 500,000 CMSIM events.
• Production of 150,000 fully reconstructed ORCA events.
• Production of ooDigis (50,000 events)
• (UCSD) Installed and Tested NPACI developed Rocks Cluster Management Software
• GDMP Stability Testing - replication of 300GB of Minimum Bias events from CERN to Caltech and UCSD Tier2s.
• (UCSD) Installation and Testing of prototype Fermilab Developed Production Tools.
• Installation of Hickey Scheduler that serves processors on Caltech and SDSC Tier2.
• Installation of PBS.

Tier2 Personnel

Apart from the individuals in the following list, who are concerned with various management aspects of the Tier2, we have a growing community of end users.

Role of Other Universities in California and Elsewhere

All of the CMS member universities in California will be connected at high speed by the Calren-2 project.  We expected that UC Davis, UCLA, and UC Riverside would be the first users of this prototype center: several physicists from those institutes are already active on the Tier2. In addition there are of course CMS users from at UCSD and Caltech.  We expect that all of these institutes will have significant software and physics efforts next year (2002), when the center is operating. We intend to provide support for users from other US regions, if that turns out to be necessary despite the presence of other resources at FNAL and elsewhere. 

It may be possible, later, to enlarge the center by including resources at the other California universities.  UC Riverside is interested in this possibility. 

Tier2 FTE Effort by key personnel

This table breaks down the approximate effort by individuals at Caltech who were involved in the installation and commissioning of the Caltech Tier2.