Frequently Asked Questions for PETSc 2.0

  1. How do I begin using PETSc if the software has already been completely built and installed by someone else?

    Assuming that the PETSc libraries have been successfully built for a particular architecture and level of optimization, a new user must merely:

    a) Set the environmental variable PETSC_DIR to the full path of the PETSc home directory (for example, /home/username/petsc).

    b) Set the environmental variable PETSC_ARCH, which indicates the architecture on which PETSc will be used. For example, use "setenv PETSC_ARCH sun4". More generally, the command "setenv PETSC_ARCH `$PETSC_DIR/bin/petscarch`" can be placed in a .cshrc file if using the csh or tcsh shell. Thus, even if several machines of different types share the same filesystem, PETSC_ARCH will be set correctly when logging into any of them.

    c) Begin by copying one of the many PETSc examples (in, for example, petsc/src/sles/examples/tutorials) and its corresponding makefile.

    d) See the introductory section of the PETSc users manual for tips on documentation.

  2. The PETSc distribution is SO large. How can I reduce my disk space usage?

    a) The directory ${PETSC_DIR}/docs contains a set of HTML manual pages in for use with a browser. You can delete these pages to save about .8 Mbyte of space.

    b) The PETSc users manual is provided in PostScript and HTML formats in ${PETSC_DIR}/docs/manual.ps and ${PETSC_DIR}/docs/manual.html, respectively. Each requires several hundred kilobytes of space. You can delete either version that you do not need.

    c) The PETSc test suite contains sample output for many of the examples. These are contained in the PETSc directories ${PETSC_DIR}/src/*/examples/tutorials/output and ${PETSC_DIR}/src/*/examples/tests/output. Once you have run the test examples, you may remove all of these directories to save about 300 Kbytes of disk space.

    d) The debugging versions (BOPT=g) of the libraries are larger than the optimized versions (BOPT=O). In a pinch you can work with BOPT=O, although we do not recommend it generally because finding bugs is much easier with the BOPT=g version.

    e) you can delete bin/demos and bin/bitmaps

  3. PETSc executables are SO big and take SO long to link.

    We find this annoying as well. On most machines PETSc now uses shared libraries by default, so executables should be much smaller. Also, if you have room, compiling and linking PETSc on your machine's /tmp disk or similar local disk, rather than over the network will be much faster.

  4. Where should I send PETSc bug reports and questions?

    Send all maintenance requests to the PETSc developers via the email address petsc-maint@mcs.anl.gov . Also, see the file bugreporting. html

  5. How can I subscribe to the PETSc users mailing list?

    You can join the PETSc users mailing list by sending email to majordomo@mcs.anl.gov with the message, "subscribe petsc-users". We will update users regarding new releases, changes, etc. through this mailing list.

  6. What happened to the very cool "domain" directory that was in previous versions of PETSc and allowed me to easily set up and solve elliptic PDEs on all kinds of grids? I can't find it in PETSc 2.0.

    That code was all written only for sequential machines. We hope to redo it for parallel machines using PETSc 2.0 someday. Domain is no longer available or supported.

  7. PETSc has so many options for my program that it is hard to keep them straight.

    Running the PETSc program with the option -help will print of many of the options. To print the options that have been specified within a program, employ -optionsleft to print any options that the user specified but were not actually used by the program and all options used; this is helpful for detecting typo errors.

  8. PETSc automatically handles many of the details in parallel PDE solvers. How can I understand what is really happening within my program?

    You can use the option -log_info to get more details about the solution process. The option -log_summary provides details about the distribution of time spent in the various phases of the solution process. You can use ${PETSC_DIR}/bin/petscview, which is a Tk/Tcl utility that provides high-level visualization of the computations within a PETSc program. This tool illustrates the changing relationships among objects during program execution in the form of a dynamic icon tree.

  9. Assembling large sparse matrices takes a long time. What can I do make this process faster?

    See the Performance chapter of the users manual for many tips on this.

    a) Preallocate enough space for the sparse matrix. For example, rather than calling MatCreateSeqAIJ(comm,n,n,0,PETSC_NULL,&mat); call MatCreateSeqAIJ(comm,n,n,rowmax,PETSC_NULL,&mat); where rowmax is the maximum number of nonzeros expected per row. Or if you know the number of nonzeros per row, you can pass this information in instead of the PETSC_NULL argument. See the  manual pages for each of the MatCreateXXX() routines.

    b) Insert blocks of values into the matrix, rather than individual components. 

  10. I want to use PETSc only for uniprocessor programs. Must I still install and use a version of MPI?

    For those using PETSc as a sequential library, the software can be compiled and run without using an implementation of MPI. To do this, edit the file ${PETSC_DIR}/bmake/${PETSC_ARCH}/base.site and change the lines that define the location of MPI to MPI_LIB = ${PETSC_DIR}/lib/lib${BOPT}/${PETSC_ARCH}/libmpiuni.a MPI_INCLUDE = -I${PETSC_DIR}/src/mpiuni MPIRUN = ${PETSC_DIR}/src/mpiuni/mpirun If you compile PETSc as such, you will be able to run PETSc ONLY on one processor. Also, you will be able to run the program directly, without using the mpirun command.

  11. How do such a small group of people manage to write and maintain such a large and marvelous package as PETSc?

    a) We work very efficiently.

    1. We use Emacs for all editing; the etags feature makes navigating and changing our source code very easy.
    2. Our manual pages are generated automatically from formatted comments in the code, thus alleviating the need for creating and maintaining manual pages.
    3. We employ automatic nightly tests of PETSc on several different machine architectures. This process helps us to discover problems the day after we have introduced them rather than weeks or months later.

    b) We are very careful in our design (and are constantly revising our design) to make the package easy to use, write, and maintain.

    c) We are willing to do the grunt work of going through all the code regularly to make sure that all code conforms to our interface design. We will never keep in a bad design decision simply because changing it will require a lot of editing; we do a lot of editing.

    d) We constantly seek out and experiment with new design ideas; we retain the the useful ones and discard the rest. All of these decisions are based on practicality.

    e) Function and variable names are chosen to be very consistent throughout the software. Even the rules about capitalization are designed to make it easy to figure out the name of a particular object or routine. Our memories are terrible, so careful consistent naming puts less stress on our limited human RAM.

    f) The PETSc directory tree is carefully designed to make it easy to move throughout the entire package.

    g) Our bug reporting system, based on email to petsc-maint@mcs.anl.gov, makes it very simple to keep track of what bugs have been found and fixed. In addition, the bug report system retains an archive of all reported problems and fixes, so it is easy to refind fixes to previously discovered problems.

    h) We contain the complexity of PETSc by using object-oriented programming techniques including data encapsulation (this is why your program cannot, for example, look directly at what is inside the object Mat) and polymorphism (you call MatMult() regardless of whether your matrix is dense, sparse, parallel or sequential; you don't call a different routine for each format).

    i) We try to provide the functionality requested by our users.

    j) We never sleep.

  12. Does all the PETSc error checking and logging reduce PETSc's efficiency?

    Actually the impact is quite small. But if it really concerns you to get the absolute fastest rate you can, then edit the file ${PETSC_DIR}/bmake/${PETSC_ARCH}/base.O and remove -DPETSC_DEBUG and -DPETSC_LOG. Then recompile the package. We do not recommend this unless you have a complete running code that is well tested, and you do not plan to alter it. Our measurements never indicate more then a 3 to 5% difference in performance with all error checking and profiling compiled out of PETSc.

  13. How can I generate performance summaries with PETSc?

    Firstly, to generate PETSc timing and flop logging, the compiler flag -DPETSC_LOG (which is the default) must be specified in the file petsc/bmake/${PETSC_ARCH}/base.${BOPT} Then use these options at runtime: -log_summary -optionsleft See the Performance chapter of the users manual for information on interpreting the summary data. If using the PETSc (non)linear solvers, one can also specify -snes_view or -sles_view for a printout of solver info. Only the highest level PETSc object used needs to specify the view option.

  14. Why do you use MPI?

    MPI is the message-passing standard. Because it is a standard, it will not change over time; thus, we do not have to change PETSc every time the provider of the message-passing system decides to make an interface change. MPI was carefully designed by experts from industry, academia, and government labs to provide the highest quality performance and capability. For example, the careful design of communicators in MPI allows the easy nesting of different libraries; no other message-passing system provides this support. All of the major parallel computer vendors were involved in the design of MPI and have committed to providing quality implementations. In addition, since MPI is a standard, several different groups have already provided complete free implementations. Thus, one does not have to rely on the technical skills of one particular group to provide the message-passing libraries. Today, MPI is the only practical, portable approach to writing efficient parallel numerical software.

  15. How do I debug on the Cray T3D/T3E?

    Use TotalView. First, link your program with the additional option -Xn where n is the number of processors to use when debugging. Then run totalview programname -a your arguments The -a is used to distinguish between totalview arguments and yours.

  16. How do I debug if -start_in_debugger does not work on my machine?

    For a uniprocessor job, ex1, with MPICH using ch_p4 as the underlying communication layer, the procedure is:

    - Create a dummy file with the text "local 0" -

    - Start the debugger directly: gdb ex1

    - Run with a command such as: run -p4pg dummy

    With MPICH using shmem as the underlying communication layer, the procedure is:

    - dbx ex1 - run -np 3 (other petsc options) .

  17. Why is PETSc programmed in C, instead of Fortran or C++?

    C enables us to build data structures for storing sparse matrices, solver information, etc. in ways that Fortran simply does not allow. ANSI C is a complete standard that all modern C compilers support. The language is identical on all machines. C++ is still evolving and compilers on different machines are not identical. Using C function pointers to provide data encapsulation and polymorphism allows us to get many of the advantages of C++ without using such a large and more complicated language. It would be natural and reasonable to have coded PETSc in C++; we opted to use C instead.