Infrastructure Today
In this document, "infrastructure" refers to the cables, pathways, closets, and other spaces that we use to provide the physical support for UCSD's data, voice, wireless, and video networks.
We install Category 6 cable and Category 6 components in new buildings and in buildings we upgrade. We pull a minimum of four separate cables to every outlet in most new buildings, and in buildings where we anticipate a lot of connections (such as the Engineering complex), we pull six cables. The Next Generation Network (NGN) provides funding to upgrade all of the cabling on campus which is not at least Category 5E, and we use copper for all speeds up to 1Gbps. We install very little fiber to the desktop, and fiber that was installed is mostly still unused.
We work closely with UCSD’s department of Facilities Design and Construction on all new construction projects, and have developed good templates and written standards to ensure that buildings are cabled properly. Closet size is important, and for all new buildings we try to follow the dimensions recommended in the BICSI standards. We also try to get multiple electrical circuits (on generator backup if available), air conditioning and good grounding. All of this is sometimes difficult because of architectural and/ or budgetary issues with the buildings. In the oldest buildings on campus, some of the closets are not in good condition. They can be small and exposed to heat, dust, and atmospheric conditions. In some cases we need to install equipment in utility cores, where we have started using enclosed racks to keep as much dust as possible away from our switches. Another issue is station conduit size. Our standard is 1-inch or 1¼-inch station conduits as these allow us to pull the number and type of cables we require, and to upgrade the cabling in the future.
We install fiber to every building and to every closet. Over the years the number of fibers has increased, as has the proportion of single mode fiber to multi-mode. We usually install at least 48 strands of fiber to a new building, and large buildings get double that amount or more. The single mode we use is SMF28, the industry standard for a long time, and we use 62.5-micron multimode fiber. The demand for fiber use between buildings continues to grow dramatically. Fiber is used extensively for the main data network, the voice network, the cable television network, and specialized research networks such as the Optiputer, which presently combines multiple parallel 1 Gbps paths to accumulate its high speeds. The inter-building fiber plant is also used for circuits that provide video services for the Media Center and connections to CENIC and public carriers.
More than 330 separate fiber cables connect the buildings on campus. A database in TMS, the department’s Telemanagement System, tracks these cables and how their individual fiber strands are being used. Over the last few years the workload for turning up new circuits was very heavy and we haven't had enough resources to keep the database current.
UCSD's network is based around Node rooms. These rooms are dedicated to telecommunications equipment, and serve as cabling hub points for various areas of the campus. They contain the voice and data equipment which forms UCSD's backbone network. There are currently 15 node rooms on the main campus, and six node rooms in off-campus locations. In the past few years we have worked extensively to equip these rooms with good air conditioning, Inergen fire suppression systems, and UPS and generator power backup. In two locations we depend on a portable generator system owned by PPS. In an extended power outage we call PPS and they bring the generator and hook it up to a pre-wired connection. We are trying to install our own permanent generators at these two sites.
Campus Changes
UCSD continues to grow at a rapid pace. Even though the number of undergraduates is being reduced as a result of the state budget crisis, campus research activities are expanding rapidly. New buildings are going up, and we anticipate heavy growth at the School of Medicine, the Campus Center, and the East Campus Medical Complex. More than a million assignable square feet of new space will be built by 2010, and we expect that this will result in a demand of 15,000 to 20,000 new connections. As a reference, our network has about 50,000 connections today. Many of the construction projects being built or planned are shown on the Capital Planning Web site.
We spend a lot of resources supporting campus growth, especially when we need to do major reroutes in order to clear sites for new construction. In the future there will be large and complicated reroutes associated with the San Diego Supercomputer Center addition, the Price Center expansion, and the San Diego Trolley line.
UCSD’s construction schedule, large as it is, cannot keep pace with the growth in staff, so the campus is leasing more off-site offices. In recent months new offices have been occupied at Executive Drive and Dunhill Street, and more departments have moved staff to the La Jolla Professional Center. In order to provide fast and cost-effective services, we purchased dark fiber connections from the campus to these buildings. We have been able to partner with CENIC to get these links installed. This is a great benefit for the campus as dark fiber is almost impossible to obtain from commercial carriers.
UCSD is not only getting bigger, it is becoming more network intensive. The number of connections is rising faster than the number of employees, and networking is becoming a key element in how the campus does business. The San Diego Supercomputer Center and the new Cal(IT)2 institute are also factors driving the growth and visibility of networking at UCSD.
The increasing density of cables, and the possibility of more fiber to the desktop in future years, means we must ensure that the cable pathways in new buildings are engineered and installed properly. We have always specified 1¼-inch station conduits, although there is often pressure on us to reduce this requirement because of budgeting issues in new buildings. We also need to make sure that cable trays are large enough to accommodate future cable growth and cable upgrades.
The demand for point-to-point fiber connections around the campus continues to grow, especially in support of high-speed research networks. As speeds grow, we anticipate the need to use fusion splicing rather than connectors to build many of these links.
The Health Insurance Portability and Accountability Act of 1996 (HIPPA) means that we must spend more time and resources in safeguarding our networks, especially in areas such as the School of Medicine and the Cancer Center, where medical records are used extensively. We need to ensure good physical security and access control to our closets and node rooms, and install entry card systems, automated entry logs, and possibly video surveillance in these areas.
Technology Changes
New cabling standards are still in development. An IEEE study group is looking at the possibility of 10Gbps over unshielded copper, although it’s too early to say if this will be achievable or not. Category 7 is a standard for shielded copper cables, and is available in Europe. We don’t think it will ever be used here as it is more expensive and labor intensive to install than fiber to the desktop.
There is very little Wave Division Multiplexing on campus at present, but we see that technology helping us to use our fiber resources more effectively. Coarse Wave Division Multiplexing (CWDM) is already cost effective, and Dense Wave Division Multiplexing (DWDM) will follow, probably first in the academic research networks. Wave Division Multiplexing may also lead to increased use of fusion splicing, which has an impact on how we lay our fiber racks out in the nodes, as it uses more space.
New technologies, especially Voice Over IP (VOIP), will place new demands on our closets. VOIP requires that all closets have UPSs, and sometimes generator backup. In addition, power for the VOIP handsets is provided over a pair of copper strands in the station cable. Sometimes this is done with external power injectors, which need more space in the closet. Air conditioning and other environmental factors are also becoming more important for closets.
Large Effective Area Fiber (LEAF fiber), developed by Corning, is a new technology for long distance fiber links. Used by the carriers in their backbones, it provides better performance over long distances. LEAF fiber is relevant to UCSD because we use some of our existing SMF28 cables to connect to CENIC at the eastern edge of the campus. In the future we may need to upgrade this fiber to LEAF. Although it's still far from clear, we are monitoring the situation.
We considered a new 50-micron standard for multi-mode fiber that would work over slightly longer distances than the present 62.5-micron fiber. However, we think it will be unnecessary because it's now inexpensive to light single mode fiber, and single mode can be used for all distances beyond that of the 62.5 multimode.
We need to change the way that we think about the capacity of our networks. With new applications, particularly the distributed processing in research networks, the network needs to be built to accommodate extremely fast burst modes. In previous years networks were engineered to provide a certain throughput, but in new networks the transmission speeds are now more important than the extended traffic capacity. This means that what we must upgrade to what we previously viewed as 'over-engineered' networks, and be comfortable with the notion of 'wasting cheap bandwidth'.
Some Specific Actions
- Meet with the management at Facilities Design and Construction to update them on emerging technologies, and what that means for new buildings
Closets will need to be sized to BICSI specifications. Air-conditioning and backup power, when available, should be installed, and the cable pathways need to be large enough for future upgrades. Electrical sub-panels may also be required in some closets.
- Continue to build and upgrade our node rooms to support campus growth; in the short term, bring Node S (Center Hall) and the new node at the Cancer Center into service
- Continue to install Category 6 cables until a new standard comes out, then decide if we should adopt that standard
In the future it may make financial and technical sense to install widespread fiber to the desktop, but that point may not be within the next five years. Some academic research networks may need fiber to the desktop, and we will meet these requirements on a case-by-case basis.
- Implement cross-campus fiber use agreements with other campus departments so we can track what fiber circuits are in use, and recover the fiber when it is no longer needed
- Rebuild the fiber records database in TMS, and keep it current
- Lay out fiber racks in the nodes, to prepare for more extensive use of fusion splicing; purchase a fusion splicer so we can build DWDM links to tight specifications
- Build a physically redundant fiber path between the main campus and the East Campus Medical Complex, when the new bridge over Interstate 5 is complete
- Continue to upgrade and replace the air conditioning, UPS systems and generator backups in the node rooms; install permanent generators in the two locations where we presently rely on portable generators
- Investigate what we need to set up CWDM links
These will let us use our fiber resources more efficiently, and if combined with optical cross-connects, may also allow us to automate much of the fiber circuit provisioning. It is unlikely that CWDM is cost effective this year, but it may become so within the next five years.
- Install an access control system in all of the node rooms and closets of buildings where medical records are held, as part of our HIPPA compliance.
Impact on Resources
Many closets in the oldest campus buildings are unsuitable for new technologies such as VOIP. Upgrading these closets would be very difficult and extremely expensive. In these buildings it may be better not to deploy technologies which require pristine closets, but instead use more traditional ways of providing equivalent services.
Gigabit ethernet is becoming widely available, and is as affordable as 100 mbps ethernet was a few years ago. The Next Generation Network will install 10/100/1000 ports as standard from now on, although this was not part of the original funding model. Gigabit ethernet requires Category 5E or Category 6 cabling. Up to now, the NGN has not been upgrading Category 5 cables, so the cost of replacing these cables with Category 6 will need to built into the NGN budget for the future. The total number of Category 5 cables on campus is not known, but is probably less than 3,000.
We need more fiber technician manpower in order to handle the heavy workload and to rebuild and maintain the fiber database.
If LEAF fiber to the east campus is required, then it will be a requirement driven by research initiatives. It may be more appropriate to fund it with research funds rather with NGN funds.
Our network is growing faster than the number of Communications Users on campus. This is something that needs to be addressed when we plan the next five-year phase of the NGN funding plan.