Pole Blog '05

Gerald Przybylski (Electronics Engineer) describes the logistics of science and social events on the 60th parallel south  11/22-12/4/05

Colleagues Bill Edwards (LBNL Project Manager) Arthur Jones, Keith Beattie, and David Hays (Software Engineers) upgraded and installed new hubs and the GPS system


Tue, 22 Nov 2005 18:54:53

Subject: if I were back in Berkeley, I'd probably want to know too...

Hi all,
>Just thought I would take a minute to hit a couple of the high spots...
>The flight from ChC (Christchurch, New Zealand) to McM (McMurdo Station, Antarctica) was aboard a Kiwi Herc... with wheels...
>(I understand that Lockheed doesn't sell skier Herks outside the US)
>48 passengers. 36 up front in jump seats, a pallet of IceCube cargo
>(3 red daq crates) in the middle, and 12 in the back in airliner seats
>screwed down to a pallet. On the tail door there was another pallet
>of cargo... mostly our check bags.
>Late start... around noon. Good flight. Orientation in McMurdo, but no
>altitude sickness lecture like last year. Bag drag immediately
>after orientation. Our evening meal was midrats.
>Up early for a 7:15am muster time for the 10 min. trip out to the
>hard ice runway, right in front of McMurdo station.
>The flight to pole was good. The load-master told us we would be crossing
>the Transantarctic range about an hour after take-off. The digital cameras
>were busy taking pictures of snow, mountains and glaciers. Arrival before
>lunch time Saturday, your Friday... Orientation after lunch. Bill and I
>are in the new station building. (phone 720 568 1642 for me, and 720 568 1628
>for Bill).
>The flight included Tom Geisser, Matt Newcomb, and two Kiwi carpenters
>who will be working on the IceCube permanent counting house.
>Monday, Bill instigated an inventory of DOM Hubs. Not everything was found
>until helpful discussions, and email from Kael. Tuesday morning,
>after locating the last of our stuff, we started updating DOM Hubs.
>Unfortunately (Darryn explained) there is cargo grid-lock for "do not freeze"
>cargo all the way from pole to Christchurch. The DNF facilities are full, so
>cargo is backing up. That includes, apparently, our Rev 1 DOR cards, our
>spare disk drives, swappable disk trays for the HD drives, and our
>clock distribution and cables. We're a little worried about
>getting everything on time.
>Instead of completing a hub at a time, we will be doing what we can with the
>materials we brought with us.
>A critical defficiency is hard disk drives, because just about all the Western
>Digital (model Caviar) drives have gone bad. We can't seem to find all the
>spare Hitachi Deskstar drives we are looking for either.
>As of this evening, Claire Petterson and Adam Bouchte have a DOM Hub
>they can use to start DOM testing, once the new operating system and
>software load is installed on it. It has as many upgrades as is our goal,
>made up from spares we brought along. That hub has DOR Rev 0 cards,
>'which are perfectly fine for DOM testing.
>We are working on a second DOM Hub; it's about as far as we can take
>it, given the lack of disks, disk trays, and DOR Rev 1 cards.
>Tomorrow, more of the same.
>Arthur emailed from ChC. We are looking forward to his arrival.
>Jim Yeck may be bringing some spares we need as well...
>jerry and bill

Wed, 23 Nov 2005 19:55:39

Subject: Not a bad day today...

up early for the 5:30am teck-board call into Madison, attended by
>Tom Geisser, Bill, and I.
>After breakfast, Bill and I rode the shuttle (first day of the
>season for the shuttle van)
>to the dark sector, and walked from the drilling camp to the temporary
>counting house. There we commenced upgrades on more DOMs, and
>enhanced our record keeping.
>After lunch, Darryn called a meeting to discuss the schedule, and deliveries
>by cargo. He also announced the pending occupation of a portion of
>room B2, in the new station, by the contents of the back-of-
>science. That's a mile stone...
>After the meeting, back at the temporary counting house. Late in the
>eight crates of DNF DAQ equipment arrived. The crates included all the
>hardware we were waiting for. After unpacking some of it, we were able to
>complete an upgrade on a DOM. The first one upgraded was the IceTop hub.
>String 21, and the DOM testing hub (Claire, and Adam) will get DOR Rev 1
>cards first.
>Our days of ranting about cargo are over... Tomorrow is another day...
>jerry & bill

Thu, 24 Nov 2005 20:46:28
Subject: A day of catch-up

Between Breakfast and Lunch we sorted through some of the supplies
>that arrived yesterday.
>Darryn's 1pm meeting: Who's doing what, and what is needed to proceed...
>No word on delivery to pole of the remaining 5 boxes of DAQ...
>After the meeting, we installed DOR Rev 1's in the DOM
>hub that Clare, Adam, and Martjn will use for pre-deployment testing.
>By the afternoon, Mark, in Madison, was using the hub.
>One DOM was connected around mid-afternoon. It produced 100 KHz noise rates,
>until it was covered with black fabric. It settled down to 3 KHz after that.
>The catch-up also came in the afternoon.
>Those hubs which we had partially upgraded before the supplies arrived
>got their disk trays, Rev 1 DOR cards, and rudimentary boot checks.
>Because the DOR cards have recent firmware, and the (Western Digital)
>disk we are using for our tests hasn't had an upgrade since last January,
>the fvers.pl script delivered some error messages. At the moment,
>we are not worried.
>We were happy that all the DOR cards were recognized. Arthur will help us
>make sense of the rest.
>Arthur is on the passenger manifest for arrival tomorrow, 11/25, on schedule.
>I exchanged email with him this morning before 9am, then none subsequently.
>Ah... if they only had satellite internet on the C-130... and maybe
>an in-flight movie...
>and maybe some heat too...
>Perhaps he'll find a way of getting on the internet from McM tonight...
>The flight is most likely a Herc, so an arrival time well after dinner
>is possible.
>It's still about -33F here; sometimes the wind kicks up to 14 kts.
>Fortunately, it was only 8 kts for our walk back from the temporary
>counting house
>for dinner.
>I read in the Nov 20 issue of Antarctic Sun that a traverse team plans to
>complete the journey to the South Pole in January of '06. (this season)
>Last year a traverse team made it to the edge of the polar plateau.
>When they turned
>around, they left behind a D-8, which they'll pick up on the way
>thru this year.
>(In order to get a D-7 to pole, it needs to be partly disassembled,
>then the parts flown
>to pole for reassembly. I guess even that option isn't practical for a D-8.
>The cargo most favored for pole-bound trips is fuel, or bulky, or heavy items.
>6 tractors will bring in 420,000 kg of cargo, plus a fuel sled and
>sleeping compartments.
>The workhorse tractor is a Case model stx450. I urge you to look it
>up on the web.
>or locate a copy of the Antarctic Sun on-line. See page 11. You'll see why.
>(420,000 kg of cargo is 33 Herc flights to the pole)
>The return trip of some future traverse may carry the disassembled
>dome back to McM.
>That'll be the end of an era.
>There are still three buildings left under the dome, and lots of
>frozen food-stuffs.
>The two big ones are the science building, and the comms building.
>Enjoy your turkey. We get ours Saturday.
>Chef James Brown arrived yesterday; he's the replacement for Cookie John.
>Jerry and Bill

Fri, 25 Nov 2005 23:31:54
Subject: plunging ahead

Another glorious day in paradise... The temperature crept up to
-27F for a while. The wind varies between 8 and 14 kts. Partly cloudy in the morning, and think high clouds now.
>After breakfast, more DOM upgrading. We overcame a few obstacles
>on the way.
>We even found a fan not running due to a malfunction in the monitor
>card; it worked
>after reprogramming the monitor card jumpers, and using a different output.
>All five hubs which arrived two days ago in the DNF shipment are upgraded.
>The total, so far, is 10 completely upgraded, and one partially,
>including DOR cards.
>There are DOR cards here for 3.5 more hubs, assuming no spares. i.e.
>total of 14 complete.
>Bill has arranged for more DOR cards, and disk drives, to be hand carried here
>by a collaborator scheduled to arrive soon.
>The DOM Testers have been having a few problems which seem to be
>software related.
>They could talk to a DOM for a while, but now cannot even talk to
>one of the DOM MBs
>I brought for moat testing. Hmmm...
>I hope we can set up our DOM testing disk configuration so that when
>the hub boots,
>it recognizes the DOR 1 cards, and DOMs with new flash files.
>This may take a while, especially if we have to upgrade the OS from Redhat 9.x
>to Redhat enterprise.
>The DOR driver RPM I tried to install had a bad checksum. Either
>the file didn't download
>properly, or there is a compatibility problem requiring upgrade of the OS.
>Arthur emailed that his flight was cancelled for mechanical reasons.
>His next possible flight in will
>be Monday (Sunday your time).
>Arthur could be a great help to Claire, Adam and Martjn with their
>DOM communications
>problems... Too bad about the delay.
>Otherwise, the day hasn't been very eventful.
>Just trying to keep to task, and not make mistakes.
>Having a wonderful time, wish you were here...
>Jerry & Bill

Sat, 26 Nov 2005 18:37:05
Subject: short day; Thanksgiving celebration

This morning, after an early breakfast, John and Paul W. went to the temporary
>counting house and the TOW to move some DOM Hubs around, and finish
>up a few things. Bill finished off the updates on the hub I started last night. When
>I arrived we went over
>our spreadsheet again. The spreadsheet claimed 18 hubs, and we both
>knew there
>were only 16, counting Mark's DOM Test Hub.
>Two of the pending hubs are 100% vanilla. We do not anticipate problems.
>The spare hub may also be vanilla.
>The intermittent hub may be a time sink. We'll see what we can do.
>Our concern is mitigated by the knowledge that the CPU boards in use this year
>will be replaced by lower power CPUs next year. In future we won't have the
>concern about fan failures on the CPU chips, or gigabit ethernet chips.
>We have come to appreciate this year how big a failure category fans are.
>The spare GPS clock, unpacked, and powered up last night, never
>locked onto satellites,
>even after programming in the location and time of day. As in
>Berkeley, the antenna
>was located in a window with Pi sky coverage. i.e. in Berkeley, our
>GPS clocks lock on
>with as little as 2Pi/3 sky coverage. At pole, it appears that 2Pi
>sky coverage is mandatory.
>After building an insulated 'housing' from hi density foam
>insulation to retain the heat of the
>electronics inside the antenna, and placing the antenna on the side of the
>TOW just clear of the roof, within a few minutes, the clock acquired
>three satellites.
>GPS will be available soon for moat testing, and for the DOM tests too,
>though I suspect the STF testing of old and new DOMs doesn't really
>benefit from GPS.
>At 1pm, after Brunch, (egg on bacon on English muffin, or egg on
>spinach on English muffin,
>with smoked salmon and cream cheese for to add) we were kicked out
>of the galley so that
>the Thanksgiving tables could be set up. The first seating is at
>around 3:30pm.
>The second seating around 5:30, or 6pm, and a third seating even later....
>Non management grantees kick in $20 for wine for all, and management
>grantees kick in $50...
>Around 3:30, the second floor hallway door between the A and B pod closed.
>Musicians (winter-overs, I'm told) serenade those in the second floor A pod
>hallway who were partaking of the shrimp cocktail, baked brie in crust,
>artichoke dip, the olive plate, the cheeses... and a glass of wine from the
>bar (at the end of the galley).
>Those who had nice clothes along dressed up. There were a few neck ties,
>and one tartan kilt in our seating.
>At around 4pm, we were invited/urged to take seats at the galley
>tables which were
>rearranged in 4 long tables 2 x 2, set with white table cloths and
>cloth serviettes.
>Each place set with plate, (plastic) wine glass, (Cambro plastic)
>water glass, and (institutional) flatware.
>After a toast offered by BK Grant, and Vladimir Papitashvhili, we
>filled our plates buffet style.. (from
>the steam trays). Turkey, Smoked turkey, potatoes, gravy, peas,
>yams, stuffing, and vegetarian
>stuffing, bread, and salad with vinaigrette (There was a BIG raid on
>the hydroponic garden).
>Apple pie, pumpkin pie, strawberry cheese cake, or pecan pie were
>offered as dessert,
>with coffee to finish. Our plates were bussed for us too!
>Just when we were settling into conversation we were politely urged
>to forsake our seats
>for the preparation for the next seating. The second seating list,
>about 100 places, filled up
>rapidly. Those of us who were late got the first seating, mostly,
>except those who took the
>third seating, presumably for scheduling reasons. The third seating
>wasn't posted.
>Serving a posh dinner for 245, three or four thousand miles from
>civilization is no small feat.
>I salute the kitchen staff, and volunteers, that made it possible.
>jerry and bill

Mon, 28 Nov 2005 02:55:56
Subject: short day, today...

James Brown, the new head chef, made our omelettes this morning.
>Ham, green pepper, onion, olive, green chili, cheese... and a great cranberry coffee-cake. Sausage patties, and small boiled potatoes too. Orange juice (reconstituted), and a great view of the south pole marker out the window. Another day in paradise.
>After brunch, Bill and I felt like walking off some of our
>Thanksgiving dinner with
>a trip to the TOW. There, we completed the upgrades on the
>remaining DOM Hubs,
>except for the one which was reported to be intermittent. We'll
>leave it for later.
>We also ran out of DOR cards to fill this last hub, as well as the 1
>foot long, yellow
>cables that interconnect the DSB and DOR cards.
>The cables we can make up, the DOR cards are on the way...
>I hope Arthur can help us thrash the intermittent hub.
>If we can force it to crash, then we will know we are on to something.
>Was it a disk problem, or a CPU problem, or a memory problem, or
>an ethernet controller problem???...
>The clock fan-outs both work on all outputs.
>We have the time to match up the varying lengths of green fan-out cable
>to the skew of the various ports. This level of matching is hardly necessary,
>but, as I said... we have the time to do it.
>The installation of additional racks in the temporary counting house is
>moving along. They will soon be ready for our timing distribution cables.
>Since the uninterruptable power supplies change this year, different
>power sockets have to be installed in the power outlets...
>We heard there was a party that lasted 'til 4am last night...
>... but then... what happens at the pole... stays at the pole...:-)
>If the two flights hold to schedule, Arthur's flight should arrive
>before lunch.
>Len Shulman, from Delaware, is also listed... for IceCube this time,
>since CosRay funding (at pole?) came to an end. He's been a regular here for
>many, many, many years...
>Bill and Jerry

Mon, 28 Nov 2005 23:44:20
Subject: Arthur arrived; things are great!

> Arthur arrived before lunch, saw the orientation video, and greeted  us when we returned to the galley for lunch.    Fortunately for us, he felt well enough to walk out to the TOW after lunch... even shunning automated transport!!!

Magically, for the DOM Hub that wasn't communicating with DOMs yesterday, when Arthur logged in,   it suddenly worked...

Well... actually, we had to reconnect the DOM MB  that were disconnected from the filter, and placed in boxes on the shelf yesterday while we were away from the TOW...Perhaps disturbing things set something right.

I learned that the slightly out of date firmware in some of the DOR cards could have been causing our problems. With fourteen DOM Hubs in the fourteen slots in the the temporary counting house,

it looks like we have just about achieved our primary mission.

That provides for 12 icecube strings (12 hubs), and 64 icetop optical modules (2 hubs)

In the broken hub, #15,  we noted that neither of the CPU fans spins up to full speed, and the fan on the giggabit ethernet chip just jitters.  Overheating could cause the crashes...  No certain diagnosis at this time, though...

There is something I don't understand, though.  I've heard that another two DOM hubs will be needed for the acoustic detectors.  Only the 'broken' hub, and the DOM test hub remain uninstalled in the TCH.  Perhaps additional hubs will be provided for the acoustic system.  hmmm.

The filter box problem is somewhat alleviated now that we have negotiated a three way deal.

We borrow IceTop filters from last season for a while, the DOM Test people borrow short DB-9 cables used with the IceTop filters, and we get to use the IceTop filters with the long DB-9 cables (borrowed from us by the DOM Test crew ) for running MOAT in the counting house. Permissions, and passwords have been arranged so that testing can commence soon.

Some of the DOM Hubs in the TCH have their standard data taking softwareload now.  They are just about ready for moat... except the new clock fan-out isn't installed yet... and won't be (at least not permanently) until the missing rack arrives. Before the opportunity was lost, Bill and I measured all combinations of pins and shield for string 21 quad 2 using both an ohm meter and a capacitance meter.

The measurements are remarkably symmetrical, to better than a half percent. They compare very closely with string 21 quad 3, allowing for the extra cable length increasing the capacitance a little.  At the moment, we can't see any smoking guns.

The measurements required patience, since some of them required settling time of three to five minutes.  (Arthur described the terrible conditions under whichprevious measurements were made in Jan '05, in the cold and wind)

We look forward to capturing and examining waveforms from the oscilloscope like sampling feature recently added to both the DOM and DOR firmware.

Finally... Bill and I made some timing skew measurements of both of the clock fan-out cards. I now have data on all of the six boards produced so far. It looks like the additional measurements fall into the distribution of the 4 cards previously measured.

We are ready to deploy a fan-out into the counting house as soon as the rack arrives.  Hence the crossed fingers...

So... a little progress was made today... > The rumor is that another of our DAQ crates is on the way... We hope it came on one of the two flights today.

Three flights tomorrow. We are keeping our fingers crossed.

-34.4F, with 7.5kt winds from grid 075 this evening.

GPS antenna Encased in a cavity in slabs of foam insulation. The antenna itself is about 4" above the roof. It works!! Beaker-box/IceCube-Lab in the background.

Tue, 29 Nov 2005 23:19:32
Subject: low point: GOES outage today

This morning, with Arthur's help to write a CPU intensive program, by loading both CPUs of the problem hub to 99.9%, we showed that faulty fans DO result in a system crash of DOM Hub 8. So, we replaced the CPU board, fans and all, with a fresh spare from the cabinet in the TCH.  Under the same CPU load, during an hour and  a half of running, the hub didn't crash.   We feel pretty certain we understand the problem with that hub.  (Darryn remarked that the winter-overs should have spotted that one...he said he was pretty disappointed) Darryn promised us some different fans to use with the "faulty CPU board" for a test.If the opportunity presents itself, we'd like to test the old board with functional fans to prove (or disprove) that neither the board itself, nor the (properly cooled) CPU chipsare faulty.   It's important to do this because the board we put into the hub was the only complete loaded spare board.

The other spare CPU board has neither CPU chips nor fans!!!

It's important to leave sufficient spares...apparently including fans.

All hubs now have all upgrades.  (except the DOM Test hub, which doesn't have a removable drive bay, and is running with a Western Digital disk drive.) That hub, (previously Icehub8), with four DOR Rev 1 cards, is available to DOM testing now.

Bill mounted the rack rails from spares provided by Paul W.

 At about the time the CPU problem was solved, we attempted to phone into the Monday LBL hardware meeting, only to discover that GOES was out. What bad luck...

Arthur aided Claire, Adam, and Martjn  with making sure the right releases of drivers, and firmware were installed on the DOM Test hub. They are now running smoothly with a large number of DOMs... possibly 64. This only partially describes Arthur's work today.

1) preped doms for dom tes (helping Claire and Adam out in the cold)

2) installed sw and troubleshooting for dom test Probably DOM Test Hub)

3) started testing on string 21 default comm parameters did _not_ work w/ rev1 dor cards an dor-driver 02-09-04 (open never finished default comm parameters worked marginally with my dor drive and the new 103v comm fw.  doms on card 1 and 21A and 21 had extreme retx rates in configboot in this setup o dor-driver 02-10-03 is due to be installed when the satellit comes up tomorrow morn. waveform tests will start tomorrow

4) started overnight test on icetop hub 1 (initial results look good) with jerry's dom mainboards.

More DAQ crates arrived in the afternoon... apparently all the remaining five. Two crates contained new racks. Other equipment cases contained CPUs.

The temporary counting house is quite crowded right now, with people installing, and boxes to obstruct the way.

To aid Arthur, and to get along a little farther ourselves, the old GPS clock distribution cables and passive fan-out have been removed from the temporary counting house.  The new fan-out is temporarily installed to serve the String 21 hub, and the IceTop hub under test. The new racks will have to be properly installed in their final places before we move the GPS clock and fan-out to their assigned locations.

No problems so far. The cabling will be easy. As to the near future, we heard this from Darryn:

One Friday morning at 7am IceCube will be participating in a world wide webcast called "Beyond Einstein".


Diamond dust today... The sun-dog displays were fairly persistent today, but not very spectacular any time when we checked on them.   There was only a hint of a halo.  Temperatures hovering slightly below -30F, but winds below 8 kts.

So, as you see... we're a little farther along.

jerry, Bill, and Arthur

Wed, 30 Nov 2005 23:39:48
>Subject: caught up, pretty much...

>It's the middle of the second week now...
>All the hubs are in good working order.
>5 hubs have faulty ethernet fans, but don't exhibit performance problems.
>Paul Wesniewski asked for a list of those hubs.
>Fans will be shipped to pole as soon as the
>parts and procedure become available.
>Unfortunately, replacement will have to be done
>by someone other than us, since we
>will probably leave pole before the parts arrive.
>Furthermore, if they can be procured, CPU chips
>and fans will be sent to complete
>the second spare board.
>Claire and gang are testing DOMs in groups of
>64. I've heard results are good, but
>didn't seek statistics... maybe tomorrow.
>Additional DAQ boxes arrived, including crates with the new racks.
>Many new CPUs and Uninterruptable power supplies also arrived.
>>From the drill camp, we heard that surface cable
>- to - DOR card patch cables also
>arrived. The temporary counting house isn't ready for them
>yet... not until all those new racks are moved to their permanent locations.
>The reorganization of the counting house
>disrupted Arthur's DOM Hub performance testing.
>Only 2 of 14 hubs are tested so far. When
>testing goes, it goes pretty smoothly
>We installed the spare GPS clock, and fan-out
>unit, into the rack space allocated
>for them. Cabling, again, has to wait.
>Coming full circle... during the weekly Tuesday
>LBL meeting, Bill gave a recap,
>over the speaker phone, of the past weeks activities.
>What else is new...
>It's warmed up, but the wind has picked up.
>Wind-chills are still hovering around -50°F.
>We saw a twin Otter take off today.... Oh... and a
>Crane installed the BICEP telescope into the Dark Sector Lab. Also,
>laborers are leveling the footing for the Ten
>Meter CMB telescope right next to the DSL.
>The penthouse atop DSL is also being demolished
>as a result of more sincere observation of
>building codes. It was considered a fire hazard.
>The south pole contribution to the CERN broadcast Thursday, California time,
>has been reduced to ten minutes in a half hour
>shared with other neutrino experiments
>around the world. Tom Geisser will do most of the talking in our slot.
>So, you see by the shortness of this note, it
>wasn't much of a day for news from here.
>Jerry, Bill and Arthur

Thu, 1 Dec 2005 20:42:54
>Subject: We're in

>Today, by lunch time, Paul and Karthik had all the racks moved to
>their final positions. All the replacement 30 A twist-lock power connectors were installed yesterday. So, DOM Hubs, GPS clock and fan-out could all be powered... and were.
>After lunch, we installed the distribution cables from the fan-out
>to the DOM Hubs. That allowed Arthur to resume testing.
>We reconnected String 21 to ichub6.
>A DSB in one of the new hubs appeared not to be working, so we removed
>the hub to the TOW to investigate. The fuses were OK... so the
>board was replaced
>with a spare from the cabinet. That one did work.
>We had enough time today to further investigate the faulty fans.
>This decision arose from a discussion of the probability that two CPU fans (of ten usedlast season)
>on one board should fail has a small probability... something like 1%.
>The CPU fans were mounted on a different CPU board, and tested; they
>didn't work there either.
>Arthur noticed high friction holding back back the blades... and
>also noted that the
>label said they were ball-bearing fans! So... why did they die??
>We don't know.
>Perhaps the manufacturer got a bad batch of lubricant that
>eventually dried out, leaving a gummy residue.
>Arthur spent considerable time trying to get the DSO (oscilloscope)
>mode running
>on the DOR cards on the string 21 hub. In the end, the feature was abandoned
>in order to permit the acquisition of data that sometimes might be
>corrupt, instead of always perfect.
>Keith, and Dave arrived on the 1pm flight with Jim Yeck. Jim
>dropped off, in the
>TCH, stuff we had asked Wisconsin to send along. He brought us a Filter box,
>and some spare single channel filters. Arthur now has a filter just for MOAT
>testing. Now we can return the 2km filters to IceTop.
>All the DOM Hubs now have their operating systems loaded, as well as our
>custom software.
>So, it was another quiet day...
>Remember, the CERN physics webcast is tomorrow.
>Jerry, Bill and Arthur...

Fri, 2 Dec 2005 21:59:32
>Subject: More ups, and downs

This morning spoke Tom Geisser in the CERN Einstein webcast segment
>on the IceCube
>Neutrino detector. My job was to describe the DOM in thirty seconds. The
>live feed over the conference telephone wasn't working, so spole
>could only hear
>questions by waiting the webcast latency time. It got confusing.
>After about fifteen minutes, our segment finished, so we could get
>our breakfast.
>After breakfast, we showed Dave and Keith the TOW and the changes in the
>TCH. They brought along extra hard disks, "Y" cables, and much
>desired DOR cards.
>One of those DOR cards immediately found service as a replacement for
>card 4 in ichub03. Something went wrong with the power circuit for
>channel 0 in that card.
>Replacing the card restored the hub to service
>Now that the clock fan-out wiring is in place in the counting house, I took
>oscilloscope screen snapshots of the hub-to-hub skew, and jitter measured
>for the 10MHz clock at each the test point of each DSB.
>Jitter is pretty much unmeasurable. Triggering noise or triggering threshold
>noise makes the triggered channel envelope just as wide as the
>non-triggered channel.
>Jitter in the non-triggered cannel (the DSB test point) would
>broaden the line. It doesn't.
>The span of skews is about 0.75ns. I think we can live with that.
>For some unknown reason, this year the measurements are less noisy. Jitter
>looked particularly bad last year.
>Arthur has sassessed the performance of exactly half of the DOM Hubs.
>They are ready for strings.
>He measured the pair to pair cross-talk at the DOR end of string
>21 using Kalle's comm-scope:DOR. The results were very much like the
>measurements at SPCTS in Madison. No smoking gun there.
>The problem in quad 2 must be at the DOM end of the cable.
>His testing of comm-scope:DOM has been rocky, but one, mostly
>clear, path to data taking was found. Stay tuned! The script to
>collect many
>quad 2 waveforms is running through the night.
>Outside, many DOMs were moved today.
>I suspect that Claire and Adam have gotten through another batch of DOMs.
>Things are moving along. That includes snow, and a number of 320 lb
>boxes of DOMs.
>Keith's laptop disk drive failed during the trip south. It had a
>copy of the 3 Gig CVS
>repository for local use. Recovery will cost a bit of inbound
>satellite bandwidth.
>The TCH is much neater, and better organized, now that all the new
>equipment is in the racks.
>Most of the boxes are empty. Two racks will return to Wisconsin;
>they are in the way a little.
>It's warmed up to -17F today. The wind kicked up to 20 kts, though.
>A -40 wind chill is the best I've seen so far.
>I walked around the DSL today. Many materials for the base are on site.
>They are ready to spread out the big wooden, wall-like, things upon which
>the base of the instrument. They spread the load of the heavy frame over
>a wide area.
>Jerry Bill, and Arthur

Sun, 4 Dec 2005 03:16:30
Subject: [lbnl] winding down

Today Bill composed a two page information transfer document to share with the winter-overs.   We hit upon all the difficulties we had this summer, and the helpful information we thought would carry them through the winter.

In the afternoon, we invited all there winter-overs to have a chat, rather than dumping the document on them later.   The meeting went well.   Notes were taken, particularly by Victoria.

The spare GPS antenna is now removed from the TOW, and relocated to its box atop the electronics cabinet.  The third shelf of the locker has most of the DOM Hub spares.

Arthur found good values for comm parameters for all of string 21.

They are higher  than for the cable at SPCTS.  The captured comm

 waveform streams contain features that suggest ringing on the pulses. The ringing appears to be undersampled (above the nyquist frequency)...  Several gigabytes of string 21 communications waveform data will take a while to analyze.   Several other break-outs exhibit the ringing symptom. On all quads, including quad 2, the cross-talk is quite small, observed at the DOM end of the cable.

The wind blew fiercely  today, sometimes over 20 kts.  The temperature was in the mid teens, but felt colder.  More diamond dust iced crystals in the air today too. (e.g. http://www.sundog.clara.co.uk/halo/common.htm)

The snow drifted a great deal today.  The drillers are tough, but even they reacted to the wind, and cold.

I heard one IceTop tank got filled. (rumor...)

Three batches of DOMs have been tested so far, by Claire and crew...

It's Saturday... They served Pizza.  IceCubers were responsible for the dish-washing this evening.  It's a concession to our high population here.

> Jerry, Bill and Arthur

Sun, 4 Dec 2005 23:39:20
>Subject: checking out... passing the torch...

Bill and I leave on the first flight, which arrives at about 11:30,
>and probably departs at about noon.
>We spent an hour or two sorting between what should stay as the LBNL tool kit,
>vs what we should carry home, vs. what cargo will return to Berkeley
>in a couple of months via vessel.
>Today, Paul and Karthik moved in some more new UPS's.
>The counting house looks organized, except for the racks to be shipped back
>to UW.
>Arthur made a first pass test of all DOM Hubs demonstrating that they function
>as normal DOM Hubs when talking to DOM Main Boards. No surprises.
>For the string 21 effort, he got a modified DOM FPGA code from Thorsten
>which tries make the commscope:DOM work better. It's getting closer, but
>not golden yet. We have commscope data from all but the most troublesome
>pairs, like Q2. There are still demodulation issues.
>For a quad with one troublesome pair, and one good pair, there is
>only baseline
>crosstalk between pairs, either way. It doesn't matter how much traffic is
>present on the other pair in the quad, or in any other quad... there
>is just negligible
>crosstalk. On the other hand, any ringing that occurs for one DOM
>of a particular
>pair also occurs for the other DOM. That means it's not an x5
>amplifier malfunction.
>It suggests that it is anywhere between the COMDAC driver on one DOM MB
>and the COMDAC driver on the other DOM MB. It could also be something
>upstream in the cable, but near by, or in the 17 meters of cable,
>plus connectors,
>between DOMs. For the moment, we're a little stumped.
>We heard that the DAQ was running from DOM to event builder up north,
>and that pieces of it are, or soon will be tested in the TCH. The
>timing is very good,
>because the counting house really looks ready for it. Racks full of servers,
>and DOM Hubs.
>Sixteen or so DOMs are mounted in IceTop tanks now.
>We still haven't seen the lost water buffalo... the one mistakenly
>marked for the Palmer, or
>for Palmer Station in ChC.
>Still strong winds keeping diamond dust in the sky. Some pretty
>distinct halos,
>with sun dogs and that rainbow arc right overhead. Temperatures
>above -20F, though.
>Wind chill around -40F. The snow crunches differently under your boot.
>We heard the Disco party last night went 'til 5am or so. Not much disco after
>the first couple of songs... Heavy emphasis on party...
>Farshid tended bar in the galley. G&T and Martinis... and good conversation.
>Jerry, Bill, Arthur, Dave & Keith

Back to Top