NetBeans Architecture Answers for Bootstrap module

General Information

Question (arch-what): What is this project good for?
Answer: The NetBeans launcher starts the application. Different launchers are provided for different supported platforms - Unix/Linux, Windows, and others. There are also a handful of classes in the org.netbeans and org.netbeans.core packages which directly support early startup features, such as command-line options.
Question (arch-overall): Describe the overall architecture.
Answer:

The heart of NetBeans Runtime Container. Handles lifecycle of installed modules and OSGi bundles.

Question (arch-usecases): Describe the main use cases of the new API. Who will use it under what circumstances? What kind of code would typically need to be written to use the module?
Answer:

Invalidating Caches

NetBeans Module system is optimized to eliminate useless I/O operations on start (as a result the embedded OSGi container is fastest on the planet -- with respect to application start up time). This is achieved by caching information known to have been needed in previous starts and using it rather than obtaining it again from the module JAR files.

There are various caches (like all-layers.dat, all-manifests.dat, all-resources.dat, netigso-bundles, etc. ) provided by the module system or by other subsystems of the application. Together, working in orchestration, they eliminate the need to touch the disk (which is very slow operation especially during morning launch). Btw. there is a test which verifies the caching system really works -- it is recommended for each product built on top of NetBeans Platform to copy and adjust it to verify the caches are really used.

Of course, the caches may get out of date, for example when an external tool modifies the installation layout. How can the system detect whether the caches are valid and still be usable? The only way to verify the caches are 100% correct is to regenerate them and compare whether the cached bits are the same. However that would be terribly slow and defeat the whole purpose of using caches.

As such we have the lastModified API. Every cluster (as enumerated by netbeans.dirs and netbeans.home properties) is supposed to contain such file. Whenever an external tool changes something inside some cluster, it is supposed to touch the file and change its timestamp. That will tell the system that caches are invalid (as their timestamps will become older than newest .lastModified file).

Generating Shared Cache

The caches (as introduced in invalidating caches section) are optimized to reflect the state of previous start. However that means, the very first start runs without them. This may not be a problem (if the start follows immediately after installation), however in multi user environment (when the installation is done by administrator), the first start may be slow.

To mitigate that the system offers support for shared caches. As part of installation, one can also copy certain cache files into the shared location. Those files will then be used on a first start of the system (when the user directory is empty).

To generate the shared cache files start NetBeans with an empty, temporary user directory and then copy the desired files into first cluster directory:

$ netbeans --userdir /tmp/nb -J-Dnetbeans.close=true -J-Dorg.netbeans.core.WindowSystem.show=false
$ cd /tmp/nb
$ zip -r $INSTALL/$first_netbeans_dirs_dir/var/cache/populate.zip var/cache/netigso
$ cp var/cache/* $INSTALL/$first_netbeans_dirs_dir/var/cache
$ rm -r /tmp/nb/
         

The meaning of populate.zip and list of known cache files is described here. Additional modules and subsystems may add new files however. The ultimate knowledge is available only to those who understand overall product installation structure.

Question (arch-time): What are the time estimates of the work?
Answer:

XXX no answer for arch-time

Question (arch-quality): How will the quality of your code be tested and how are future regressions going to be prevented?
Answer:

XXX no answer for arch-quality

Question (arch-where): Where one can find sources for your module?
Answer:

The sources for the module are in the NetBeans Mercurial repositories.

Project and platform dependencies

Question (dep-nb): What other NetBeans projects and modules does this one depend on?
Answer: The core/startup module depends on FileSystems, Modules, Utilities and core/bootstrap.
Question (dep-non-nb): What other projects outside NetBeans does this one depend on?
Answer: None.
Question (dep-platform): On which platforms does your module run? Does it run in the same way on each?
Answer:

Unix (incl. Linux)

The nbexec launcher is fully featured and supported. It should work on any Unix flavor. Implemented as a shell script.

lock-file - The lock file ${netbeans.user}/lock will have file permissions changed to go-rwx for safety on Unix systems (requires presence of chmod)

Windows (all variants?)

The nbexec.exe launcher is fully featured and supported. Implemented using Visual C++; prebuilt binary is kept in CVS along with sources.

Question (dep-jre): Which version of JRE do you need (1.2, 1.3, 1.4, etc.)?
Answer: 1.4 or later.
Question (dep-jrejdk): Do you require the JDK or is the JRE enough?
Answer: The launcher itself does not use the JDK. The Unix and windows launchers permit either the JDK or JRE to be specified - if the JRE is specified, modules with hard dependencies on JDK tools (e.g. the JPDA Debugger module) will simply not be loaded. For other launchers, unknown.

Deployment

Question (deploy-jar): Do you deploy just module JAR file(s) or other files as well?
Answer: Consists of various launcher scripts in the lib/ directory of the NetBeans installation. Bootstrap Java code (part of core) resides in lib/boot.jar and core/core.jar.
Question (deploy-nbm): Can you deploy an NBM via the Update Center?
Answer: As part of core. But not on Windows.
Question (deploy-shared): Do you need to be installed in the shared location only, or in the user directory only, or can your module be installed anywhere?
Answer: Shared location only, for obvious reasons.
Question (deploy-packages): Are packages of your module made inaccessible by not declaring them public?
Answer: N/A for the launcher itself. For bootstrap code: no; the core execution engine (core-execution.jar) needs access to org.netbeans.TopSecurityManager. Modules may use org.netbeans.CLIHandler to add command-line option handling (currently used by utilities).
Question (deploy-dependencies): What do other modules need to do to declare a dependency on this one, in addition to or instead of the normal module dependency declaration (e.g. tokens to require)?
Answer:

XXX no answer for deploy-dependencies

Compatibility with environment

Question (compat-i18n): Is your module correctly internationalized?
Answer: Yes, except for early-startup code (including the native launcher) that deals with the command line.
Question (compat-standards): Does the module implement or define any standards? Is the implementation exact or does it deviate somehow?
Answer: No.
Question (compat-version): Can your module coexist with earlier and future versions of itself? Can you correctly read all old settings? Will future versions be able to read your current settings? Can you read or politely ignore settings stored by a future version?
Answer: N/A
Question (compat-deprecation): How the introduction of your project influences functionality provided by previous version of the product?
Answer:

XXX no answer for compat-deprecation

Access to resources

Question (resources-file): Does your module use java.io.File directly?
Answer: Yes, it must. The Filesystems API has not been loaded when it runs. .lastModified -

To speedup testing that module caches are up-to-date, the system recognizes special file inside of each cluster. If .lastModified file is present in a cluster, then the content of the cluster is not scanned any more deeply and instead the time stamp of the file is used as last modification for the whole content of the cluster.

This API is primarily intended for use by installer and RPM packagers and also autoupdate, that are supposed to create such file, as soon as they finish modifications to any cluster.

This file is ignored when accessing user directory, as it is expected that user can modify it, while the user usually does not have access permision or need to modify the shared installation clusters.

cached.lastModified -

If the .lastModified file is not present in cluster directory, it is recomputed on startup and stored in $userdir/var/cache/lastModified/$clustername. This file and its content is meant just for private consumption of the IDE.

installation.cache -

Installer can speed NetBeans IDE start by generating caches when the installation is over. Such caches are stored in the installation, in first cluster listed by netbeans.dirs and have the same layout as user caches (including here in described caches like all-resources.dat, as well as caches provided by other subsystems like netigso-bundles) in the user directory - e.g. they are under var/cache.

If a file var/cache/populate.zip is found, its content is unzipped into user directory (if it is empty).

all-resources.dat -

To speedup class loading during startup, the system keeps a cache of resources needed during startup in var/cache/all-resources.dat. If the cache is valid, it is loaded during startup and class and resource loading requests are redirected there.

The file is binary and shouldn't be modified by other code than org.netbeans.Archive. The file content uses magic header and versioning to allow evolution of its (private) format. The location is however well known for purposes of installer.

all-files.dat -

Rather than seeking on disk and using access OS call uselessly to find that some file is not present, the layout of files under the clusters is recorded in var/cache/all-files.data.

The file is binary and its format is private. The location is however well known for purposes of installer.

all-layers.dat -

Modules register their functionality into the system using layers. To avoid parsing and merging of content of these files on each start, the system creates a cached, binary representation of the layers structure in var/cache/all-layers.dat and var/cache/all-local-layers.dat.

The file is binary and its format is private. The location is however well known for purposes of installer.

all-clusters.dat -

Lists relative paths of clusters (those defined by netbeans.home and netbeans.dirs) for which the cache has been created. This is an important file in the shared cache as it helps to ensure the set of clusters has not changed since the cache has been generated.

The file is binary and its format is private. The location is however well known for purposes of installer.

all-installer.dat -

Additional information about modules (like deprecation message, etc.) is stored in var/cache/all-installer.dat to avoid the need to parse it out on each start.

The file is binary and its format is private. The location is however well known for purposes of installer.

all-manifests.dat -

Graph of dependencies between modules, sorted startup order, state of modules is stored in var/cache/all-manifests.dat to avoid the need to compute this information on every start.

The file is binary and its format is private. The location is however well known for purposes of installer.

all-modules.dat -

Pre-parsed content of config/Modules/*.xml is cached in var/cache/all-modules.dat.

The file is binary and its format is private. The location is however well known for purposes of installer.

package-attrs.dat -

Cache of various attributes of packages (vendor, spec & impl version, etc.) is stored in var/cache/package-attrs.dat. This file is often almost empty to signal such information is not provided by any module.

The file is binary and its format is private. The location is however well known for purposes of installer.

localeVariants -

File var/cache/localeVariants contains information about used branding and L10N files for each module. Avoids checking for locale/xyz_cs_CZ.jar files when (for example) only English L10N are present.

The file is binary and its format is private. The location is however well known for purposes of installer.

Question (resources-layer): Does your module provide own layer? Does it create any files or folders in it? What it is trying to communicate by that and with which components?
Answer: No.
Question (resources-read): Does your module read any resources from layers? For what purpose?
Answer: No.
Question (resources-mask): Does your module mask/hide/override any resources provided by other modules in their layers?
Answer: No.
Question (resources-preferences): Does your module uses preferences via Preferences API? Does your module use NbPreferences or or regular JDK Preferences ? Does it read, write or both ? Does it share preferences with other modules ? If so, then why ?
Answer:

XXX no answer for resources-preferences

Lookup of components

Question (lookup-lookup): Does your module use org.openide.util.Lookup or any similar technology to find any components to communicate with? Which ones?
Answer: lookup.org.netbeans.CLIHandler -

Instances of org.netbeans.CLIHandler are found using services lookup in the "dynamic classpath loader" (i.e. startup classpath plus ${netbeans.home}/core/*.jar).

org.netbeans.ModuleFactory -

Instances of org.netbeans.ModuleFactory are found using services lookup in the "dynamic classpath loader" (i.e. startup classpath plus ${netbeans.home}/core/*.jar). ModuleFactory can be used to create alternative module implementations.

CoreBridge -

The communication between core.jar and rest of the platform code in org-netbeans-core.jar is handled thru handled thru CoreBridge calls.

RunLevel -

Additional callbacks from core.jar after the module system is started are done using RunLevel interface. Currently it starts window system.

java.util.logging.Handler -

There is a communication bridge between the default logging handler registered by core/startup and the UI handler in core that presents certain log records in the UI.

Question (lookup-register): Do you register anything into lookup for other code to find?
Answer: No.
Question (lookup-remove): Do you remove entries of other modules from lookup?
Answer: No.

Execution Environment

Question (exec-property): Is execution of your code influenced by any environment or Java system (System.getProperty) property? On a similar note, is there something interesting that you pass to java.util.logging.Logger? Or do you observe what others log?
Answer:
• JDK_HOME - Default JDK installation directory to use, unless --jdkhome is given. [Unix only]
• JAVA_PATH - Backup variable to check for JDK installation directory. [Unix only]
• NB_JDK_HOME - Overriding JDK installation directory specifically for NetBeans. [Unix only]
• JavaDevelopmentKit - The registry key Software\JavaSoft\Java Development Kit is the default JDK installation directory to use, unless --jdkhome is given. [Windows only]
• netbeans.mainclass - Main Java class for bootstrap code to start (currently org.netbeans.core.startup.Main). Might be used by alternate launchers.
• NetBeans-IDE-Dev - The registry key Software\netbeans.org\NetBeans IDE\Dev Default user directory, unless --userdir is given. The name of this registry key is brandable, as is its default value. [Windows only]
• netbeans.home - NetBeans installation directory (where the launcher is).
• netbeans.user.dir - If this property is provided, it is used as the current working directory regardless of the value of user.dir.
• netbeans.dirs - Additional secondary NetBeans installation directories as proposed in installation structure design document.
• netbeans.classpath - Classpath to prepend to core loader (normally just lib/*.jar). Possibly useful for testing infrastructure etc.
• netbeans.systemclassloader.patches - By setting this system property to a classpath (list of directories and JARs separated by the normal path separator) you may append to the system class loader.
• netbeans.patches.cnb - By setting a system property to "netbeans.patches." + module.getCodeNameBase() one can influence installed modules without changing the build. Format is -Dnetbeans.patches.org.nb.mods.foo=/path/to.file.jar:/path/to/dir.
• org.netbeans.ProxyClassLoader - This logger logs at WARNING level about accessing resources from the default package.
• netbeans.importclass - There is a special support for importing the settings from previous versions. As only the product itself knows about what to import and where this cannot be done directly in the launcher, but we need at a well defined moment (user directory is missing and no modules have been initialized yet) to call the product to ask the user and do the actual copy of userdir.
  After all core/*.jar files has been initialized and the user dir has not yet been updated (see bellow) the launcher checks for value of netbeans.importclass system property and if provided it loads that class and invokes its main method (in AWT thread) and if no exception is thrown it marks the userdir as already upgraded. imported-marker - ${netbeans.user}/var/imported is used to identify whether a userdir has already been updated or it still needs an update. Can be created by installer if no update check should be performed, no other code is supposed to realy on this file. netbeans.accept_license_class - Before first usage of IDE user must accept license. If user does not install IDE by installer (user must accept license during installation) user must accept license during IDE first start.

  Launcher calls method showLicensePanel of netbeans.acceptlicenseclass if license was not yet accepted by user and it is necessary to show license in dialog to user.

  . license-check-marker-file - ${netbeans.user}/var/license_accepted and ${nbcluster.dir}/var/license_accepted are used to identify whether user accepted license.
• netbeans.logger.console -

  Controls whether to log messages to the console, or just to the log file.

• org.netbeans.CLIHandler -

  Setting this property to less than zero value enables logging of what is going on in the CLIHandler - e.g. in the code that locks user directory and handles processing of command line options. Use -J-Dorg.netbeans.CLIHandler=-1 to send the logging to System.err.

• org.netbeans.CLIHandler.server -

  One can disable the CLI server (listening on commands from subsequent invocations) by setting property org.netbeans.CLIHandler.server to false.

• org.netbeans.core.systemfilesystem.custom -

  Contains class name of a class that can serve as provider of the writable layer of the system file system. The class must be a subclass of org.openide.filesystems.FileSystem and have public default constructor. Morevoer the class denoted by the property must be on the classpath.

• netbeans.security.nocheck - By default NetBeans prevent certain operations like System.exit. One can disable this behavior by providing -Dnetbeans.security.nocheck=true. Since version 2.47 such property also allows the SecurityManager to be replaced by another.
• sun.awt.datatransfer.timeout - Specifies the amount of milliseconds the call to getContent() method of ExClipboard waits to synchronize with system clipboard. Defaults to 1000ms.

There are also some options passed to the launcher (interpreted either by the script itself, or by early startup Java code). Options parsed by plugins using org.netbeans.CLIHandler (e.g. -open FILE) are not included in this list:

• --jdkhome - JDK (or perhaps JRE) installation directory.
• --clusters - The list of clusters to intialize the system from. Is used to set the value of netbeans.dirs.
• --userdir - User directory, if not the default.
• -JWHATEVER - Pass option WHATEVER to the JVM.
• --cp - -cp:a and -cp:p append and prepend to the startup classpath; deprecated except for special purposes such as installing a custom look & feel.
• --ui - Specify a look & feel. The same as --laf. To make human life easier simple translation was added to convert L&F ID to L&F class name for known L&Fs.
  • Metal is translated to javax.swing.plaf.metal.MetalLookAndFeel
  • GTK is translated to com.sun.java.swing.plaf.gtk.GTKLookAndFeel
  • Nimbus is translated to com.sun.java.swing.plaf.nimbus.NimbusLookAndFeel
  • Windows is translated to com.sun.java.swing.plaf.windows.WindowsLookAndFeel
  • Aqua is translated to apple.laf.AquaLookAndFeel
• --laf - Specify a look & feel. The same as --ui.
• --bootclass - Alternative main class used to initialize the system.
• --fontsize - Specify base fontsize. Deprecated in favor of using themes which is much more general. But this is a convenient quick settings, handy for projection screens and so on.
• --locale - Specify system locale.
• --branding - Specify application branding.
• --nologging - Do not write a log file.
• --nosplash - Do not show the splash screen.
• --nogui - Do not show any GUI at all.
• --reload - Reload a test module.

Some of the branded keys may also be of some interest, they are enumerated under the property group, but all of them are currently part of org/netbeans/core/startup/Bundle.properties file and thus can be branded by overriding them in org/netbeans/core/startup/Bundle_brandingname.properties

• SplashOnByDefault - can be true or false, allows some applications build on top of the platform to disable splash by default
Question (exec-component): Is execution of your code influenced by any (string) property of any of your components?
Answer: java.util.logging.Level.1973 - LogRecords sent to registered logger may be marked as those intended for user. This can be done by logging with level which has intValue() == 1973. Localized message of such log records is then shown in a special dialog. This behaviour is supported for backward compatibility, but in fact it is deprecated. The preferred way is to directly show dialog using DialogDisplayer. netbeans.exception.alert.min.level - Minimum integer level that triggers the blinking icon signaling an exceptional state. By default 900 - e.g. WARNING. netbeans.exception.report.min.level - Minimum integer level that triggers the dialog with exception. By default 900 with assertions on and 1001 without them.
Question (exec-ant-tasks): Do you define or register any ant tasks that other can use?
Answer:

XXX no answer for exec-ant-tasks

Question (exec-classloader): Does your code create its own class loader(s)?
Answer: ClassPath-Composition - Bootstrap code creates a class loader to load the rest of the core from. The bootstrap code is loaded from the Java application class loader, using the classpath lib/*.jar.
Dynamic-ClassPath-Composition - After creating bootstrap classpath another low level set of JARs is loaded with a new classloader from core/*.jar directories in each clusters.
Question (exec-reflection): Does your code use Java Reflection to execute other code?
Answer:

The bootstrap code of course uses reflection to start the normal core. Currently some reflection is used in bootstrap class loaders to work around a variety of JRE bugs in handling loading from JAR files, especially on Windows which has mandatory file locking - the JRE code to load from JARs is rather buggy when it comes to changing or removing the JAR during the JVM session.

Javeleon -

When present as a Java agent on the boot classpath, Javeleon is invoked reflectively to enhance module reloading.

Question (exec-privateaccess): Are you aware of any other parts of the system calling some of your methods by reflection?
Answer: None known.
Question (exec-process): Do you execute an external process from your module? How do you ensure that the result is the same on different platforms? Do you parse output? Do you depend on result code?
Answer: No, except for the updater tool (part of Auto Update, q.v.). chmod is run if available to disable read access for other users to ${netbeans.user}/lock; no output is parsed but the result code is checked (if the executable in fact exists).
Question (exec-introspection): Does your module use any kind of runtime type information (instanceof, work with java.lang.Class, etc.)?
Answer:

ProxyClassLoader (the general NetBeans class loader which loads all of core and modules) checks to see if its parent loaders include instances of ProxyClassLoader, in which case it delegates to them differently. This is necessary due to the design of that loader.

ExceptionAnnotatableUsingLogRecords - To support classification of exceptions and also annotation of exceptions with logging levels and additional localized messages, logged exceptions and their initCause's are searched for implementation of Callable<LogRecord[]>. If an exception implements this interface, the call() method is called and returned LogRecords then scanned for messages, levels, etc.

Question (exec-threading): What threading models, if any, does your module adhere to? How the project behaves with respect to threading?
Answer:

XXX no answer for exec-threading

Question (security-policy): Does your functionality require modifications to the standard policy file?
Answer:

XXX no answer for security-policy

Question (security-grant): Does your code grant additional rights to some other code?
Answer:

XXX no answer for security-grant

Format of files and protocols

Question (format-types): Which protocols and file formats (if any) does your module read or write on disk, or transmit or receive over the network? Do you generate an ant build script? Can it be edited and modified?
Answer: The bin/ide.cfg (or ~/ide.cfg, Unix only) configuration file is read if present. This consists of zero or more valid launcher options, separated by spaces and/or newlines.
Question (format-dnd): Which protocols (if any) does your code understand during Drag & Drop?
Answer: N/A
Question (format-clipboard): Which data flavors (if any) does your code read from or insert to the clipboard (by access to clipboard on means calling methods on java.awt.datatransfer.Transferable?
Answer: N/A

Performance and Scalability

Question (perf-startup): Does your module run any code on startup?
Answer: Of course - NetBeans!
Question (perf-exit): Does your module run any code on exit?
Answer: When the launcher finds that the JVM has exited, it may run core/updater.jar if there are available NBM updates downloaded.
Question (perf-scale): Which external criteria influence the performance of your program (size of file in editor, number of files in menu, in source directory, etc.) and how well your code scales?
Answer: N/A
Question (perf-limit): Are there any hard-coded or practical limits in the number or size of elements your code can handle?
Answer: No.
Question (perf-mem): How much memory does your component consume? Estimate with a relation to the number of windows, etc.
Answer: Trivial.
Question (perf-wakeup): Does any piece of your code wake up periodically and do something even when the system is otherwise idle (no user interaction)?
Answer: No.
Question (perf-progress): Does your module execute any long-running tasks?
Answer: N/A
Question (perf-huge_dialogs): Does your module contain any dialogs or wizards with a large number of GUI controls such as combo boxes, lists, trees, or text areas?
Answer: No.
Question (perf-menus): Does your module use dynamically updated context menus, or context-sensitive actions with complicated and slow enablement logic?
Answer: No.
Question (perf-spi): How the performance of the plugged in code will be enforced?
Answer:

XXX no answer for perf-spi