Webservers, Webmail & Diagnosing AFS Problems

Tracy J. Di Marco White

gendalia@iastate.edu

Iowa State University

  • 26,000 active student accounts
  • 10,000 active staff and faculty accounts
  • 1 GB default quota
  • quota rental at 0.56 / 100MB / day ($1.70/GB/month), $1 minimum bill
  • Iowa State University Web presence

    www.iastate.edu

    Iowa State University Web presence

    www.public.iastate.edu

    Iowa State University Web presence

    webmail.iastate.edu

    Iowa State University Web presence

    webdev.its.iastate.edu

    Iowa State University AFS Servers

    The Problem

    We have a lot of highly visible stuff in AFS

    Monitoring

    Log files

    Sometimes you can find incidental bugs this way, like an unsigned int that should really be afs_uint32 (thanks to developer help)

    Core files & Dumping other information

    Creating a core dump

    SIGXCPU will create client.dump, hosts.dump & callback.dump

    Gathering data for use in debugging

    tcpdump

    Gathering data for use in debugging

    rxdebug against a client has helped to determine when a new OpenAFS client was talking to an old Transarc AFS server, which was causing it to throw away its tokens.

    cmdebug against a client has helped to determine that a particular volume was causing problems, leading us to notice that a disk had failed, when the operating system hadn't noticed yet

    Gathering data for use in debugging

    'rxdebug server 7000' will contain information about connected clients
    Connection from host 10.10.201.116, port 7001, Cuid b27eaca6/11a81228
      serial 406, natMTU 1260, security index 0, client conn

         call 0: # 203, state dally, mode: receiving, flags: receive_done

         call 1: # 0, state not initialized

         call 2: # 0, state not initialized

         call 3: # 0, state not initialized

    Gathering data for use in debugging

    % rxdebug 10.10.201.116 7001 -version
    Trying 10.10.201.116 (port 7001):

    AFS version: OpenAFS1.4.0008

    % cmdebug -server 10.10.201.116 -addr
    UUID: 1f404ac9-ab91-40e2-a7-94-3d416440dd45
    Host interfaces:
    10.10.201.116, netmask 255.255.255.0, MTU 1260
    Capabilities:
    Error Translation

    Gathering data for use in debugging

    Connection from host 10.10.154.207, port 25796, Cuid b27eaca6/11a99280
      serial 2, natMTU 1260, security index 0, client conn

         call 0: # 1, state dally, mode: receiving, flags: receive_done

         call 1: # 0, state not initialized

         call 2: # 0, state not initialized

         call 3: # 0, state not initialized

    Further information about rxdebug

    "state" describes the state of the connection.

    "mode" describes the mode of the connection.

    further information about rxdebug

    "flags" describe the flags of the connection

    information about cmdebug

    Links to more information

    Thanks

    I would not have learned nearly as much, nor had most of these problems solved, without the help of, and instructions from:

    And of course, everyone who works on AFS has my thanks for their continued work.

    Questions?