README-devel file for GASNet https://gasnet.lbl.gov This is a design document for GASNet. Anyone planning on modifying or adding to the GASNet code base should read this file. Other documentation: * For GASNet user documentation, see README * For GASNet licensing and usage terms, see license.txt. * For documentation on a particular GASNet conduit, see the README file in the conduit directory (also installed as README- in the docs directory). * For documentation on the communication-independent GASNet-tools library, see README-tools. * Additional information, including the GASNet specification and our bug tracking database, is available from https://gasnet.lbl.gov * Anyone planning to modify or add to the GASNet code base should also read the following developer documents, available in the GASNet git repository, which can be browsed online: https://bitbucket.org/berkeleylab/gasnet/src/develop + README-git: Rules developers are expected to follow when committing Contents of this file: * GASNet Design Information * System Organization * GASNet coding standards * GASNet porting guide * Source Control and GASNet GASNet Design Information ========================= See the GASNet specification for information about the GASNet API. A copy is included with this archive in docs/, and the definitive version is located on the GASNet webpage here: https://gasnet.lbl.gov/ The GASNet API is a two-level interface consisting of the lower-level "core API" and the upper-level "extended API". All of the functions in the extended API can be implemented using the core API, but some implementations may choose to implement them directly for performance reasons. We use the term "conduit" to refer to any complete implementation of the GASNet API (both layers), which targets a specific network device or lower-level networking layer. A conduit is comprised of any required headers, source files and supporting libraries necessary to provide the functionality of the GASNet API to GASNet clients. Some conduits may be monolithic - implementing the entire GASNet API directly on the lower-level network system, and others may be layered - directly implementing just the GASNet core API and borrowing a reference implementation of the extended API which is written using the core primitives. This distribution provides several conduits for different networks that are packaged together for convenience because they share some code and general infrastructure. However, each conduit is logically a stand-alone implementation of the GASNet API. Note there may eventually be other GASNet conduits (e.g. provided by vendors) which need not be included as part of this distribution - all that's required of a conduit is that it implements the standardized GASNet API. System Organization =================== The directory tree for this distribution looks something like this: top-level/ \__ config-aux/ \__ tests/ \__ docs/ \__ extended-ref/ \__ {conduitname1}-conduit/ \__ {conduitname2}-conduit/ \__ template-conduit/ \__ other/ * top-level: contains the GASNet configuration scripts and master Makefile and GASNet headers. When making changes to GASNet (esp the configure script), please remember that GASNet is not only a communication system for UPC, but also a self-contained product used as the communication system for other languages (like Titanium) so the configuration scripts, header files, etc. should not rely on anything from outside directories not included in this distribution (other than basic system headers and lower-level networking libraries). * Header files: The GASNet master header file (gasnetex.h) #includes several files which should be provided by each conduit: includes the following files: Created by the configure script, this header helps in customizing gasnet for site-specific and platform-specific information. Defines various basic and optimizing compiler macros, platform-independent integer types and c++ binding macros. Extended and core API platform independent helpers (providing services which aren't part of GASNet API, but are required by code in gasnetex.h) Implements GASNet's sophisticated internal tracing and statistical collection system definitions of basic constants and types used by the core API and included early in the gasnetex.h file (gasnetex.h provides some default values for various constants and types that can be overridden here) Allows a specific extended implementation to override the default gasnet types such as handles and threadinfo. Provides the public interface for the GASNet core API (function prototypes and/or inline functions for implementing the core API). .. This is the core's entry point to provide access to private symbols which might be required by the core header (e.g.: node's node id, number of nodes in job), but which should not really be publicly visible to client code (i.e. not part of GASNet API) Provides the public interface for the GASNet extended API .. Extended API's entry point to define private symbols used by the extended header (e.g.segment info, thread info) but which should not really be publicly visible to client code (i.e. not part of GASNet API) Many conduits also have and files which are _not_ included by gasnetex.h (and therefore not visible to client code) and are exclusively for use in building the conduit libraries. is the conduit-independent version of these files at the top level. Internal GASNet implementation should include instead of . In GASNet-EX, and are public headers providing the gex_Coll_* and gex_VIS_* APIs (for collectives and non-contiguous RMA respectively). Internal code must never include these headers directly. Internal code requiring the definitions contained in these headers must define one or both of the following, as appropriate, prior to inclusion of gasnet_internal.h: #define GASNETI_NEED_GASNET_COLL_H 1 #define GASNETI_NEED_GASNET_VIS_H 1 is header providing a set of conduit-independent general system utilities which are not specified by the GASNet spec. The header can and is used independently of libgasnet*.a (even on systems with no supported conduits) - all that's required is a successful configure and #include . See README-tools for documentation on the GASNet Tools interface. * config-aux: helper scripts for configure * {conduitname}-conduit: the complete implementation for a particular GASNet conduit. This directory is in charge of providing/installing the headers listed above, compiling any appropriate tests and building/installing the following libraries: libgasnet-conduitname-seq.a - GASNET_SEQ implementation of GASNet libgasnet-conduitname-par.a - GASNET_PAR implementation of GASNet libgasnet-conduitname-parsync.a - GASNET_PARSYNC implementation of GASNet * template-conduit: A conduit template which should be used as a starting point for new conduit implementations. See the README.template file in this directory for detailed instructions. * other: A container directory for any external supporting libraries or utilities used by one or more conduits. * extended-ref: the reference implementation of the GASNet extended API. Conduits can optionally use some or all of the files in this directory by compiling the sources into the conduit library and installing the headers into the appropriate conduit include directory. See mpi-conduit for an example of how to use the reference extended API implementation (the template-conduit is setup to use the extended-ref implementation by default). * tests: basic tests written to the GASNet API that should run on any GASNet conduit. These can be sucked in and compiled in each conduit directory (see mpi-conduit for an example). * docs: documentation on GASNet and various conduits GASNet coding standards ======================= All GASNet conduit code is required to meet the following coding standards: * Naming Conventions for macros and identifiers: - All entry points required by GASNet spec are lower-case identifiers with the prefix gasnet_ or gex_ - All constants or macros required by GASNet spec are upper-case and preceded with the prefix GASNET_ or GEX_ - All private symbols in the reference extended API implementation are prefixed with gasnete_ (or GASNETE_ for macros) - All private symbols in a core API implementation are prefixed with gasnetc_ (or GASNETC_ for macros) - All private symbols shared throughout GASNet are prefixed with gasneti_ (or GASNETI_ for macros) - All GASNET_ or GASNETI_ configuration selection macros are either #defined to 1 or #undef'd * All non-normative identifiers appearing in public header files should use either one of the GASNet internal name-space prefixes from the previous rule, or a _ prefix to reduce the possibility of identifier name capture (eg -Dfoo= before the #include of gasnet.h). This especially applies to macro/function names, function arguments, local variables declared in macros or inline functions, type/enum names, and struct field names. This rule was added in July 2016 and applies to all new code. * GASNet code should be as close to C89 as possible (i.e. never differing without good reason), because many C compilers still do not provide full C99 support, and differ quite widely in the supported features. The following is the COMPLETE list of features introduced by C99 we PERMIT: 1. C++/C99-style // comments 2. Mid-block declarations (mixing declarations and code) 3. Conditional expression declarations, only in for loop headers, ie: for (int i=...) { ..i.. } 4. 64-bit integer literals with explicit (U)LL suffix All GASNet code should otherwise adhere to C89 restrictions. Most notably, we PROHIBIT: (incomplete list) - Do not use variable-length array declarations - Do not use zero-length structure fields (aka flexible array member) - Do not use compound literals or named struct field initializers - Do not use printf format specifiers added in C99, such as: ll,j,z,t,L,lf (Use portable_inttypes.h to get (u)int{64,ptr}_t and PRI[ud]{64,PTR,SZ,PD}) - Do not use long long types explicitly, or assume that LL suffix constants are exactly 64-bit. If a particular type width is required then use an intW_t fixed-width type declaration and/or cast. - Do not include any header files introduced by C99 or any features they introduce - Do not use restrict, __func__ or _Pragma/#pragma directly (use GASNETI_RESTRICT, GASNETI_CURRENT_FUNCTION, and the various attribute wrapper macros defined in gasnet_basic.h using configure detection) - Do not use other GNU or vendor extensions (unless properly #ifdef protected w/ fallback) * GASNet code should use full prototype declarations whenever possible: NO: extern void foo(); YES: extern void foo(void); Compiling with GCC and -Wstrict-prototypes will give you warnings for violations of this rule. * Switch statements in GASNet code should annotate any instances of "fallthough" with GASNETI_FALLTHROUGH. Here "fallthrough" is defined as a statement (within a switch) that is followed by "case ...:" or "default:" without an intervening "break". The annotation goes where the "missing break" would otherwise have appeared. Example: switch (i) { case 0: foo(); GASNETI_FALLTHROUGH case 1: bar(); } * All code in public header files should be C++98/C++14-compliant to support C++ GASNet clients. This means ensuring any macros or inline function code passes the C++98 and C++14 type checker. Note the top-level public headers already enforce extern "C" scope around the entire GASNet public headers. It is especially important for code in public headers to avoid the few dozen C constructs which are not present in the C subset of the C++98/C++14 standards. Most notably: (incomplete list) - Do not use C++ keywords in identifiers - Do not use long long types or printf specifiers (use PRI[ud]{64,PTR}) - Do not use tentative definitions - Do not use empty preprocessor function macro arguments FOO(,,,) - Do not use 'register' storage class - String literals need a trailing space before another identifier (ie "foo" BAR not "foo"BAR) See bug 3296 for a complete list. Run a "make testcxx" to verify compliance. * GASNet conduit code should never call any of the following functions directly: malloc, calloc, realloc, free, str(n)dup, assert, sched_yield, getenv, (un)setenv, mmap, pthread_* Instead, you should use the gasneti_* wrapper functions of the same name. * GASNet code may leak "permanent" data structures with a lifetime that extends through gasnet_exit(), such as frequently re-used structures kept on freelists. Any such dynamic allocation should be annotated with calls to gasneti_leak() or gasneti_leak_aligned() to preserve the value of GASNET_MALLOCFILE for the identification of unintended leaks. * The calls gasneti_malloc(), _calloc and _realloc all return NULL for zero-byte allocations. Correspondingly, gasneti_free() and gasneti_realloc() accept NULL for the pointer argument, and GASNet code should therefore NOT make an unnecessary check for NULL before calling these functions. * The calls gasneti_malloc(), _calloc, _realloc and str(n)dup all check for NULL return, and GASNet code should therefore NOT make a redundant check. The call gasneti_malloc_allowfail() exists for the rare occasion when the caller is prepared to respond to a NULL return with something better than a fatal error. However, as noted above, a NULL return is expected for a zero-byte allocation. * Use gasneti_fatalerror() to signal fatal errors - the arguments work just like printf. NEVER use "gasneti_assert(0)" (it will be compiled away), and gasneti_fatalerror() with a useful message is always preferable to abort(). * Conditional compilation for platform-specific features should be done using the standardized platform macros provided by other/gasnet_portable_platform.h, not ad-hoc vendor or compiler-provided preprocessor macros. * Code should never make assumptions about integer data type sizes. In cases that call for a specific size integral type, the (u)int(sz)_t types provided by other/portable_inttypes.h should always be used (e.g. "uint64_t" = unsigned 64-bit int). * Do NOT make assumptions about the width of values stored in type "gasneti_atomic_t". On some platforms the width may be neither 32 nor 64 bits. See README-tools for complete documentation. * Conduit code may use any of the GASNet tools documented in README-tools, but should use the gasneti_ version when referring to these tools in public headers, because gasnet_tools.h is not included by gasnetex.h. * Learn how to use the tracing macros and use them to trace interesting events in your conduit. They're invaluable for debugging mysterious behavior. See the comments in gasnet_trace.h for usage information. * Do NOT make trivial and/or non-meaningful changes to the code or makefile framework inherited from template-conduit or cloned from the extended-ref directory (e.g. re-indenting lines, moving code into different files, renaming existing identifiers, etc.) as this makes it more difficult to patch your conduit if a bug is discovered in template-conduit or a GASNet-wide change is required. * Do NOT use tab characters in GASNet code files. Use spaces to ensure correct formatting in any editor. Emacs users should: (setq-default indent-tabs-mode nil) Vim users should: :set expandtab * Do NOT commit gratuitous changes to whitespace, even to remove tab characters. Such changes have no semantic content and can make use of revision history more difficult. You can preview your changes with the following two commands: git diff -u | wc -l git diff -u -b | wc -l and if the outputs differ, then your changes include lines that have changed in only the whitespace characters. * Do not make any conduit-specific changes to the files in the extended-ref or tests directories. They do not have any Makefiles because they just provide a centralized place for files used in other directories. * When implementing the extended API for your conduit, please clone only the necessary files from the extended-ref directory that you need to change (and continue to include the unchanged files) to reduce maintenance headache (this applies especially to gasnet_extended.h, gasnet_extended_help.h, and gasnet_extended_common.c. which should never be cloned). * Makefiles should never use GNU-make specific features (vanilla BSD make only). Makefile commands and configure scripts should adhere strictly to simple Bourne shell usage (not bash). Currently there are some violations of this rule, but these should NOT be taken as license to create additional dependencies on GNU-make. * Perl code should only use features and modules available in all installations of Perl 5.005+. * Test code should follow these standards and also those listed in tests/README Only the tests are currently using GNU-make specific features, and we'd stop doing so if we could. GASNet porting guide ==================== This section documents the list of changes usually required when porting GASNet to a new, UNIX-like operating system, CPU and/or C compiler. Porting existing GASNet conduits to a new system is usually a 1-2 day effort. Writing a new conduit is more complicated (and is not covered in this section), although the "fill-in-the-blanks" template-conduit framework is very helpful - given high degree of familiarity with the target network hardware, a functional conduit can usually be constructed within a week. Subsequent tuning can take up to several months, depending on the level of NIC hardware support available to be exploited. Required porting steps: * For a new OS, add OS detection to configure.in and acconfig.h, with appropriate settings for GASNET_THREAD_DEFINES and GASNET_THREAD_LIBS. If the system supports MPI, add appropriate configure.in defaults for MPI_CC, MPI_CFLAGS, MPI_LIBS and MPIRUN_CMD. * For a new C/C++ compiler, add a detection check for the new compiler to acinclude.m4:GASNET_FAMILY_CACHE_CHECK. Add configure.in settings for MISC_CFLAGS and MISC_CXXFLAGS if appropriate. * For a new CPU, OS or C/C++ compiler, update gasnet_portable_platform.h to create an appropriate canonical platform identifier for that component. You will need the canonical identifier(s) for most of the steps that follow. Make sure to also increment the version number at the top of that header. * For a new C/C++ compiler, implement GASNETI_ASM for inline assembly in gasnet_asm.h, if supported by the compiler. Ensure configure inline modifier detection worked, otherwise tweak it in gasnet_basic.h. * For a new CPU, implement atomic increment/decrement operations in gasnet_atomic_bits.h, or enable GASNETI_USE_GENERIC_ATOMICOPS if the CPU does not support atomic operations. Implement atomic compare-and-swap, if supported by the system. For an outline of the yet-to-be-written "Atomic OPS HOWTO" see https://gasnet-bugs.lbl.gov/bugzilla/show_bug.cgi?id=1607 * For a new CPU, implement appropriate memory barriers in gasnet_membar.h: gasneti_local_wmb and gasneti_local_rmb (one or both may be no-ops, depending on the memory consistency semantics enforced by hardware). * Fix any misc compile errors that arise. * Run the gasnet tests for all supported conduits (gmake run-tests-seq run-tests-par). Pay special attention to testtools, which tests the functionality above and is the most likely to fail on a new system. Optional porting steps: * Implement GASNet high-performance timers (gasneti_tick_t) in gasnet_timer.h, if supported by the system. * Implement a new conduit for the network hardware (see template-conduit/README.template). Source Control and GASNet ========================= If you are using GASNet in your own project and wish to track any local changes you make, then you'll probably keep a copy of the GASNet sources in your project's Source Control repository (CVS, svn, Hg, Git, etc.). If so, you should be aware of potentially undesirable interactions between timestamps as maintained by your Source Control software and the files generated by GASNet's Bootstrap script. The issues are well described here: http://www.gnu.org/software/hello/manual/automake/CVS.html To summarize the link above, you must decide whether or not to commit to Source Control those files generated by GASNet's Bootstrap script. Each approach has arguments for and against. We do not dictate that users of GASNet must use one approach or the other, and we provide the tools to implement either approach. * If you keep Bootstrap-generated files under Source Control You need a mechanism to ensure that the files you checkout will be treated as being up-to-date by the rules in GASNet's Makefiles, regardless of what your Source Control software has done with timestamps. Running ./Bootstrap -T will touch the generated files in an order that ensures that they appear to be up-to-date. Caution: This is intended to correct for lost timestamp information when you retrieve files. Using "./Bootstrap -T" in the presence of locally modified versions of Bootstrap's input files will prevent those changes from being propagated to the dependent files. Inputs include configure.in, and all acinclude.m4 and Makefile.am files (and possibly others). * If you omit Bootstrap-generated files from Source Control You need a way to identify the files generated by Bootstrap to ignore or delete them. Running ./unBootstrap will remove the generated files. One may examine the contents of unBootstrap to identify files that Source Control software may be configured to ignore. Note that the set of Bootstrap-generated files may change from one release of GASNet to the next, but unBootstrap will be kept current in each release. -------------------------------------------------------------------------- The canonical version of this document is located here: https://bitbucket.org/berkeleylab/gasnet/src/develop/README-devel For more information, please email: gasnet-devel@lbl.gov or visit the GASNet home page at: https://gasnet.lbl.gov --------------------------------------------------------------------------