SecureNet Transition

ESNet UpdateJoint Techs Meeting, July 19, 2004

William E. Johnston, ESnet Dept. Head and Senior Scientist

R. P. Singh, Federal Project Manager

Michael S. Collins, Stan Kluz,Joseph Burrescia, and James V. Gagliardi, ESnet Leads

and the ESnet Team

Lawrence Berkeley National Laboratory

TWC

JGI

SNLL

LBNL

SLAC

YUCCA MT

BECHTEL

PNNL

LIGO

INEEL

LANL

SNLA

Allied

Signal

PANTEX

ARM

KCP

NOAA

OSTI

ORAU

SRS

ORNL

JLAB

PPPL

ANL-DC

INEEL-DC

ORAU-DC

LLNL/LANL-DC

MIT

ANL

BNL

FNAL

AMES

4xLAB-DC

NERSC

NREL

ALB

HUB

LLNL

DOE-ALB

SDSC

Japan

GTN&NNSA

International (high speed)

OC192 (10G/s optical)

OC48 (2.5 Gb/s optical)

Gigabit Ethernet (1 Gb/s)

OC12 ATM (622 Mb/s)

OC12

OC3 (155 Mb/s)

T3 (45 Mb/s)

T1-T3

T1 (1 Mb/s)

Office Of Science Sponsored (22)

NNSA Sponsored (12)

Joint Sponsored (3)

Other Sponsored (NSF LIGO, NOAA)

Laboratory Sponsored (6)

QWEST

ATM

42 end user sites

ESnet IP

GEANT

- Germany

- France

- Italy

- UK

- etc

Sinet (Japan)

Japan – Russia(BINP)

CA*net4

CERN

MREN

Netherlands

Russia

StarTap

Taiwan

(ASCC)

CA*net4

KDDI (Japan)

France

Switzerland

Taiwan

(TANet2)

Australia

CA*net4

Taiwan

(TANet2)

Singaren

ESnet core: Packet over SONET Optical Ringand Hubs

IPv6: backbone and numerous peers

ELP HUB

SNV HUB

CHI HUB

NYC HUB

ATL HUB

DC HUB

peering points

MAE-E

Starlight

Chi NAP

Fix-W

PAIX-W

MAE-W

NY-NAP

PAIX-E

Euqinix

PNWG

SEA HUB

ESnet Connects DOE Facilities and Collaborators

hubs

SNV HUB

Abilene

STARLIGHT

MAE-E

NY-NAP

PAIX-E

LBNL

ESnet Logical InfrastructureConnects the DOE Community With its Collaborators

ESnet Peering(connections toother networks)

Commercial

NYC HUBS

SEA HUB

Japan

SNV HUB

MAE-W

FIX-W

PAIX-W

26 PEERS

CA*net4

CERN

MREN

Netherlands

Russia

StarTap

Taiwan

(ASCC)

Abilene +

7 Universities

22 PEERS

MAX GPOP

GEANT

- Germany

- France

- Italy

- UK

- etc

SInet (Japan)

KEK

Japan – Russia (BINP)

Australia

CA*net4

Taiwan

(TANet2)

Singaren

20 PEERS

3 PEERS

LANL

TECHnet

2 PEERS

39 PEERS

CENIC

SDSC

PNW-GPOP

CalREN2

CHI NAP

Distributed 6TAP

19 Peers

2 PEERS

KDDI (Japan)

France

EQX-ASH

1 PEER

5 PEERS

ESnet provides complete access to the Internet bymanaging the full complement of Global Internetroutes (about 150,000) at 10 general/commercialpeering points + high-speed peerings w/ Abilene andthe international networks.

ATL HUB

University

International

Commercial

Abilene

EQX-SJ

Abilene

6 PEERS

Abilene

ESnet New Architecture Goal

•MAN rings provide dual site and hub connectivity

•A second backbone ring will multiply connect theMAN rings to protect against hub failure

Europe

Asia-Pacific

ESnetCore/Backbone

New York (AOA)

Chicago (CHI)

Sunnyvale (SNV)

Atlanta (ATL)

Washington, DC (DC)

El Paso (ELP)

DOE sites

NERSC

LBNL

JointGenomeInstitute

SLAC

SF BayArea

Qwest /ESnet hub

mini ring

SF BA MAN ringtopology – phase 1

Existing ESnetCore Ring

Chicago

El Paso

First Step: SF Bay Area ESnet MAN Ring

•Increased reliability and siteconnection bandwidth

•Phase 1

oConnects the primary Office ofScience Labs in a MAN ring

•Phase 2

oLLNL, SNL, andUC Merced

•Ring should not connectdirectly into ESnet SNV hub(still working on physicalrouting for this)

•Have not yet identified bothlegs of the mini ring

NLR /UltraScienceNet

Seattle andChicago

LA and San Diego

Level 3hub

Traffic Growth Continues

Annual growth in the pastfive years has increasedfrom 1.7x annually to justover 2.0x annually.

TBytes/Month

ESnet Monthly Accepted Traffic

ESnet is currently transporting about 250 terabytes/mo.

Traffic coming into ESnet = Green

Traffic leaving ESnet = Blue

Traffic between sites

% = of total ingress or egress traffic

Note that more that 90% of the ESnet traffic isOSC traffic

ESnet Appropriate Use Policy (AUP)

All ESnet traffic must originate and/or terminate onan ESnet an site (no transit traffic is allowed)

Who Generates Traffic, and Where Does it Go?

ESnet Inter-Sector Traffic Summary,Jan 2003 / Feb 2004 (1.7X overall traffic increase, 1.9X OSC increase) (the international traffic is increasing due to BABAR at SLAC and the LHC tier 1 centers atFNAL and BNL)

Peering Points

Commercial

R&E (mostlyuniversities)

International

21/14%

17/10%

9/26%

14/12%

10/13%

4/6%

ESnet

~25/18%

DOE collaborator traffic, inc.data

72/68%

53/49%

DOE is a net supplierof data becauseDOE facilities areused by universitiesand commercialentities, as well as byDOE researchers

DOE sites

ESnet Top 20 Data Flows, 24 hrs., 2004-04-20

$C:\Documents and Settings\WEJ\Desktop\TF_GUI_Main_files\TF_TopTraffic_data\TF_TopFlowGraph.png$

Fermilab (US)  CERN

SLAC (US)  IN2P3 (FR)

1 terabyte/day

SLAC (US)  INFN Padva (IT)

Fermilab (US)  U. Chicago (US)

CEBAF (US)  IN2P3 (FR)

INFN Padva (IT)  SLAC (US)

U. Toronto (CA)  Fermilab (US)

DFN-WiN (DE)  SLAC (US)

DOE Lab  DOE Lab

SLAC (US)  JANET (UK)

Fermilab (US)  JANET (UK)

Argonne (US)  Level3 (US)

Argonne  SURFnet (NL)

IN2P3 (FR)  SLAC (US)

Fermilab (US)  INFN Padva (IT)

A small numberof scienceusers accountfor a significantfraction of allESnet traffic

Top 50 Traffic Flows Monitoring – 24hr2 Int’l and 2 Commercial Peering Points

10 flows> 100 GBy/day

More than 50flows> 10 GBy/day

LBNL

PPPL

BNL

AMES

RemoteEngineer

• partial duplicateinfrastructure

DNS

Remote Engineer

•partial duplicateinfrastructure

TWC

Remote

Engineer

Disaster Recovery and Stability

•The network must be kept available even if, e.g., the West Coastis disabled by a massive earthquake, etc.

ATL HUB

SEA HUB

ALB

HUB

NYC HUBS

DC HUB

ELP HUB

CHI HUB

SNV HUB

Duplicate Infrastructure

Currently deploying fullreplication of the NOCdatabases and serversand Science Servicesdatabases in the NYCQwest carrier hub

Engineers, 24x7 NetworkOperations Center, generatorbacked power

•Spectrum (net mgmt system)

•DNS (name – IP addresstranslation)

•Eng database

•Load database

•Config database

•Public and private Web

•E-mail (server and archive)

•PKI cert. repository andrevocation lists

•collaboratory authorizationservice

Reliable operation of the network involves

•remote Network Operation Centers (3)

•replicated support infrastructure

•generator backed UPS power at all criticalnetwork and infrastructure locations

•high physical security for all equipment

•non-interruptible core - ESnet coreoperated without interruption through

oN. Calif. Power blackout of 2000

othe 9/11/2001 attacks, and

othe Sept., 2003 NE States power blackout

Disaster Recovery and Stability

•Duplicate NOC infrastructure to AoA hub in twophases, complete by end of the year

o9 servers – dns, www, www-eng and noc5 (eng.databases), radius, aprisma (net monitoring), tts (troubletickets), pki-ldp (certificates), mail

Maintaining Science Mission Critical Infrastructurein the Face of Cyberattack

•A Phased Response to Cyberattack is being implemented toprotects the network and the ESnet sites

•The phased response ranges from blocking certain site trafficto a complete isolation of the network which allows the sitesto continue communicating among themselves in the face ofthe most virulent attacks

oSeparates ESnet core routing functionality from external Internetconnections by means of a “peering” router that can have a policydifferent from the core routers

oProvide a rate limited path to the external Internet that will insure site-to-site communication during an external denial of service attack

oProvide “lifeline” connectivity for downloading of patches, exchange ofe-mail and viewing web pages (i.e.; e-mail, dns, http, https, ssh, etc.)with the external Internet prior to full isolation of the network

Phased Response to Cyberattack

LBNL

ESnet

router

borderrouter

peeringrouter

Lab

gatewayrouter

ESnet secondresponse – filter trafficfrom outside of ESnet

Lab firstresponse – filterincoming trafficat their ESnetgateway router

ESnet third response – shut down themain peering paths and provide onlylimited bandwidth paths for specific“lifeline” services

peeringrouter

gatewayrouter

border router

router

attacktraffic

ESnet first response –filters to assist a site

Sapphire/Slammer worm infection created a Gb/s of trafficon the ESnet core until filters were put in place (both intoand out of sites) to damp it out.

Phased Response to Cyberattack

Phased Response toCyberattack

Architecture to allow

•phased response to cybersecurityattacks

•lifeline communications duringlockdown conditions.

Design the Architecture

Software; site, core and peeringrouters topology, and; hardwareconfiguration

1Q04

Design and test lifelinefilters

Configuration of filters specified

4Q04

Configure and test fail-overand filters

Fail-over configuration is successful

4Q04

In production

The backbone and peering routershave a cyberattack defensiveconfiguration

1Q05

Grid Middleware Services

•ESnet is the natural provider for some “scienceservices” – services that support the practice ofscience

oESnet is trusted, persistent, and has a large (almostcomprehensive within DOE) user base

oESnet has the facilities to provide reliable access and highavailability through assured network access to replicatedservices at geographically diverse locations

However, service must be scalable in the sense that as itsuser base grows, ESnet interaction with the users doesnot grow (otherwise not practical for a small organizationlike ESnet to operate)

Grid Middleware Requirements (DOE Workshop)

•A DOE workshop examined science driven requirements fornetwork and middleware and identified twelve high prioritymiddleware services (see www.es.net/#research)

•Some of these services have a central managementcomponent and some do not

•Most of the services that have central management fit thecriteria for ESnet support. These include, for example

oProduction, federated RADIUS authentication service

oPKI federation services

oVirtual Organization Management services to manage organizationmembership, member attributes and privileges

oLong-term PKI key and proxy credential management

oEnd-to-end monitoring for Grid / distributed application debugging andtuning

oSome form of authorization service (e.g. based on RADIUS)

oKnowledge management services that have the characteristics of anESnet service are also likely to be important (future)

Science Services: PKI Support for Grids

•Public Key Infrastructure supports cross-site, cross-organization, and international trust relationships that permitsharing computing and data resources and other Gridservices

•DOEGrids Certification Authority service provides X.509identity certificates to support Grid authentication provides anexample of this model

oThe service requires a highly trusted provider, and requires ahigh degree of availability

oFederation: ESnet as service provider is a centralized agent fornegotiating trust relationships, e.g. with European CAs

oThe service scales by adding site based or Virtual Organizationbased Registration Agents that interact directly with the users

oSee DOEGrids CA (www.doegrids.org)

ESnet PKI Project

•DOEGrids Project Milestones

oDOEGrids CA in production June, 2003

oRetirement of initial DOE Science Grid CA (Jan 2004)

o“Black rack” installation completed for DOE Grids CA (Mar 2004)

•New Registration Authorities

oFNAL (Mar 2004)

oLCG (LHC Computing Grid) catch-all: near completion

oNCC-EPA: in progress

•Deployment of NERSC “myProxy” CA

•Grid Integrated RADIUS Authentication Fabric pilot

PKI Systems

Secure racks

Secure Data Center

Building Security

LBNL Site security

Internet

Fire Wall

Bro Intrusion Detection

Vaulted Root CA

HSM

DOEGrids Security

RAs andcertificateapplicants

Science Services: Public Key Infrastructure

•The rapidly expanding customer base of this service will soonmake it ESnet’s largest collaboration service by customercount

Registration Authorities

ANL

LBNL

ORNL

DOESG (DOE Science Grid)

ESG (Climate)

FNAL

PPDG (HEP)

Fusion Grid

iVDGL (NSF-DOE HEP collab.)

NERSC

PNNL

ESnet PKI Project (2)

•New CA initiatives:

oFusionGrid CA

oESnet SSL Server Certificate CA

oMozilla browser CA cert distribution

•Script-based enrollment

•Global Grid Forum documents

oPolicy Management Authority Charter

oOCSP (Online Certificate Status Protocol) RequirementsFor Grids

oCA Policy Profiles

Grid Integrated RADIUS Authentication Fabric

•RADIUS routing of authentication requests

•Support One-Time Password initiatives

oGateway Grid and collaborative uses: standard UI and API

oProvide secure federation point with O(n) agreements

oSupport multiple vendor / site OTP implementations

oOne token per user (SSO-like solution) for OTP

•Collaboration between ESnet, NERSC, a RADIUS appliancevendor, PNNL and ANL are also involved, others welcome

•White paper/report ~ 01 Sep 2004 to support earlyimplementers, proceed to pilot

•Project pre-proposal:http://www.doegrids.org/CA/Research/GIRAF.pdf

Collaboration Service

•H323 showing dramatic increase in usage

Grid Network Services Requirements (GGF, GHPN)

•Grid High Performance Networking Research Group,“Networking Issues of Grid Infrastructures” (draft-ggf-ghpn-netissues-3) – what networks should provide to Grids

oHigh performance transport for bulk data transfer (over 1Gb/sper flow)

oPerformance controllability to provide ad hoc quality of serviceand traffic isolation.

Dynamic Network resource allocation and reservation

oHigh availability when expensive computing or visualizationresources have been reserved

oSecurity controllability to provide a trusted and efficientcommunication environment when required

oMulticast to efficiently distribute data to group of resources.

oIntegrated wireless network and sensor networks in Gridenvironment

Priority Service

•So, practically, what can be done?

•With available tools can provide a small number ofprovisioned, bandwidth guaranteed, circuits

osecure and end-to-end (system to system)

ovarious Quality of Service possible, including minimumlatency

oa certain amount of route reliability (if redundant pathsexist in the network)

oend systems can manage these circuits as single highbandwidth paths or multiple lower bandwidth paths of (withapplication level shapers)

onon-interfering with production traffic, so aggressiveprotocols may be used

Guaranteed Bandwidth as an ESNet Service

usersystem2

usersystem1

site B

policer

site A

•will probably be service levelagreements among transitnetworks allowing for a fixedamount of priority traffic – so theresource manager does minimalchecking and no authorization

•will do policing, but only at the fullbandwidth of the serviceagreement (for self protection)

resource

manager

authorization

resource

manager

resource

manager

allocation willprobably berelatively staticand ad hoc

bandwidthbroker

•A DOE Network R&D funded project

usersystem2

Phase 1

Phase 2

Network Monitoring System

•Alarms & Data Reduction

oFrom June 2003 through April 2004 the total numberof NMS up/down alarms was 16,342 or 48.8 per day.

oPath based outage reporting automatically isolated1,448 customer relevant events during this period oran average of 4.3 per day, more than a 10 foldreduction.

oBased on total outage duration in 2004,approximately 63% of all customer relevant eventshave been categorized as either “Planned” or“Unplanned” and one of “ESnet”, “Site”, “Carrier” or“Peer”

•Gives us a better handle on availability metric

2004 Availability by Month

Unavailable Minutes

Jan. – June, 2004 – Corrected for Planned Outages(More from Mike O’Connor)

>99.9% available

<99.9%available

ESnet Abilene Measurements

•3 ESnet Participants

oLBL

oFERMI

oBNL

•3 Abilene Participants

oSDSC

oNCSU

oOSU

•We want to ensure that the ESnet/Abilene crossconnects are serving the needs of users in thescience community who are accessing DOE facilitiesand resources from universities or accessinguniversity facilities from DOE labs.

•Measurement sites in place:

•More from Joe Metzger

OWAMP One-Way Delay Tests Are Highly Sensitive

•NCSU Metro DWDM reroute adds about 350 microseconds

Fiber Re-Route

42.0

41.9

41.8

41.7

41.6

41.5

ESnet Trouble Ticket System

•TTS used to track problemreports for the Network,ECS, DOEGrids, AssetManagement, NERSC, andother services.

•Running RemedyARsystem server andOracle database on a SunUltra workstation.

•Total external ticket =11750 (1995-2004), approx.1300/year

•Total internal tickets = 1300(1999-2004), approx.250/year

Conclusions

•ESnet is an infrastructure that is critical to DOE’sscience mission and that serves all of DOE

•Focused on the Office of Science Labs

•ESnet is working on providing the DOE missionscience networking requirements with several newinitiatives and a new architecture

•QoS service is hard – but we believe that we haveenough experience to do pilot studies

•Middleware services for large numbers of users arehard – but they can be provided if careful attention ispaid to scaling