Friday, November 9, 2018

My Experience on Preparing for PTE Academic Exam


After postponing for a long time, today (2018-11-09) I finally sat for the PTE Academic English exam. As I did not spend enough time to prepare and practice for the exam, I do not expect that I will get a good score 😅. I will have to wait 5 more business days to see the exam results. Until then, I decided to record some of the things which can be helpful to other future test takers.

Planning

As I was postponing the preparation for this exam many times (as there were lot of other interesting tech stuff to learn than 'boring' English 😃), I decided to just schedule the exam without any preparation 😟. I scheduled the exam about 3 weeks ago, so I only had around 2 weekends + 3 more weekdays to prepare for the exam.
PTE official website itself has lots of valuable resources for you to prepare for the exam (See https://pearsonpte.com/preparation/ ). This website even provides free practice questions which you can download to your computer (See “Offline practice test” in https://pearsonpte.com/preparation/resources/  page). I downloaded the PTE test takers hand book and other such documents available from the official website. This website also has good introductions (including short videos) about each question type. Then I created a separate folder for each question type to store the resources related to that question type. This helped me to easily focus on each question type individually. Then I created a plan to prepare for the exam within the short time period available before the exam.
For PTE, there are two main areas you must focus.
  •  English enabling skills (Grammar, Vocabulary, Pronunciation, etc)
  • How to answer each question type

Preparation

Improving English Enabling Skills 

In addition to prepare for answering the questions, you must improve your core English enabling skills such as grammar, vocabulary (PTE exam focuses on academic related vocabulary), and pronunciation.
For grammar, I followed an online video course from Udemy website (by Anthony Kelleher). For vocabulary, I followed a pdf which contained "the top 400 words from PTE vocabulary". But I think my academic related vocabulary was already good due to the day-to-day work I do for my job and due to lot of reading I do, so you may have to spend more time to study more words targeted in PTE vocabulary.
For the pronunciation, I learned two techniques. First one is, to google whenever I find a word which I didn't know how to pronounce, and then click the "speaker"🔉 button in the Google 🔎 result page to listen how to pronounce it. The second technique was to use the Voice Recorder software available in Windows OS to record my own voice saying the specific word and then to listen to the recorded voice to compare it with actual expected pronunciation to identify whether I have to improve the pronunciation. You will be surprised to learn that although you thought you pronounce the word correctly in your mind, you pronounced it wrongly in reality (even some simple words). This second technique is very useful as it gives you immediate feedback to correct your pronunciation.

Preparing for Question Types

What I did first, was to re-order the question types by the complexity and difficulty, from most complex/difficult to least complex/easy. Then I could prioritize spending time on preparing and practicing the question types in the top of the list, and then go down the list. I identified that I actually did not have to prepare for some question types in the list, as those are very basic questions. I practiced answering those questions only during the mock tests.
"Write an essay" question type was on the top of my prioritized list, and the second one was "Describe an image" question type. Those were followed by "Re-tell a lecture", "Summarize Spoken Text", and "Summarize Written Text". From all of these prioritized question types, I struggled with "describe an image" question type the most.

Methods/Structures

To answer a question from each of these complex question types in PTE exam, it is not enough to have proper English enabling skills such as grammar, vocabulary, and pronunciation. It is essential to have a method, or a structure prepared beforehand for properly and quickly answer these questions within the given short time period. Fortunately, there are videos in YouTube that explain about these methods/structures by an excellent English PTE guru named Jay, from http://www.e2language.com website. I will list down links to these videos at the end of this post.

Practice

For practicing, I found some sample questions from different sources such as from Jay’s videos, from PTE website, and other online resources. As I did not have enough time left, I did not practice enough questions to make me feel fully confident. For example, I wrote only 3-5 essays before the PTE exam.
I also bought 2 scored mock tests from Pearson website (https://www.ptepractice.com/). Doing these scored mock tests was very useful to get ready for the actual exam.
Things to note: Although I had more than enough time when I did the mock exams, it seemed I was not careful with time during the real test. Therefore, I was not able to answer 1-2 questions in the first section. Also, it was little bit distracting due to the noise of the other test takers during the first section. From my overall experience during the preparation and facing the PTE exam, the most difficult question type for me was "Describe an image" question type. All other question types were manageable. It was very interesting experience (as this is my first non-tech exam after more than 10 years), but I am not exactly sure what will be the score. Let's hope for the best 🙏. [UPDATE: I received my PTE score report on next day. Score was better than I expected 😊]

Links

·        Question Type: “Write an essay”
o   PTE Writing Write Essay SUPER STRUCTURE  Sentence by Sentence with Jay! - https://www.youtube.com/watch?v=z_-u0fHQetQ
o   How to write a good essay Paraphrasing the question - https://www.youtube.com/watch?v=o9aVjBHEEbU  
·        Question Type: “Describe an image”
o   PTE Speaking: Describe Image | SUPER METHOD! - https://www.youtube.com/watch?v=rveo3RFK3kk
o   PTE ACADEMIC DESCRIBE IMAGE with RESPONSES - ULTIMATE COLLECTION !! - https://www.youtube.com/watch?v=T9_PYMRyQCY
·        Question Type: “Re-tell a lecture”
o   PTE Speaking  Retell Lecture METHOD with Jay! - https://www.youtube.com/watch?v=U9iYnbHd5ys
·        Question Type: “Summarize Spoken Text”
o   PTE Listening: Summarize Spoken Text | METHOD with Jay!  - https://www.youtube.com/watch?v=cc86wI4v4rc
·        Question Type: “Summarize Written Text”
o   PTE Writing: Summarize Written Text | Learn the Proven Method! - https://www.youtube.com/watch?v=AdyRgrOEzx8


~~~


Tuesday, October 30, 2018

Kaizen Singapore CTF - My first CTF experience


Today (27 of Oct, 2018), I attended a CTF organized by Div0, Booz Allen Hamilton, and ICE71, based on Kaizen CTF platform of Booz Allen Hamilton. According to Booz Allen Hamilton, they had done similar events throughout the world using this platform. I got to know about this event through Div0's meetup page (https://www.meetup.com/div-zero/events/255394149/). Although I had heard about CTFs and was interested in participating CTFs for a quite some time, this was the first CTF I felt comfortable enough to attend. Main reasons were, it was a single player CTF and organizers had specially mentioned that this was a beginner friendly CTF which can be an ice breaker of CTFs. As I did not know any other colleagues who were interested and had security skills to form a CTF team, I was not able to attend CTFs which expect teams.

As I learned, there are two types of CTFs. One type is attacker-defender type CTFs, and the other type is jeopardy style CTFs. The CTF organized by Kaizen was a jeopardy style CTF.
The event started by hosting a lunch and networking session at around 1pm, and then after a quick introduction about rules and the platform, the CTF was started at around 3pm.
The time duration for the CTF was from 3pm to around 7pm. At 3pm, the Kaizen platform allowed us to access the CTF challenges. The CTF challenges were categorized under 'coding', 'reverse engineering', 'crypto', 'web', 'networking', and 'forensics'.

As we were instructed not to share the details of the CTF challenges, I will try to give information about the experience without leaking information about the challenges.
I started the first challenge in 'coding' category, although I understood what was the expected algorithm, due to lack of experience in developing expected kind of scripts, I did not continue to work on creating a script for this. I stopped it there and then started challenges in 'Web' category. From around 6-7 challenges in 'Web category, I completed 3. I also almost completed 2 more 'Web' challenges, but I was not able to continue those two at the last step. Then I moved to complete a challenge from each 'Forensics', 'Networking' categories, and 2 challenges from 'Crypto' category.

So I completed 7 challenges and almost completed around 3 more challenges. Although my score was not that high, I think it is a good score for a first time CTF. As I was afraid I would be stuck in rabbit holes, I moved away to other challenges when I felt I was stuck at one challenge for few minutes. I am not yet sure whether that is a good approach for CTFs, or whether I should try to complete high value single tasks taking more time. My strategy was trying to complete basic challenges of each category if I feel like I have the basic knowledge, then move to work on other advanced challenges in categories familiar to me, and to move to next one if I am stuck more than few minutes.

There were very valuable prizes prepared for the top 5 players in the leader board, including 2000 SGD worth training voucher for a security training such as OSCP/OSCE for the first place winner. The organizers had organized this event excellently and only complain I have is that room was too cold at the end. It would be great if there was hot coffee or tea there. Overall it was worthy learning experience and I would like to thank Div0, Booz Allen Hamilton, and ICE71 for their effort on organizing this event.


~~~


Thursday, April 20, 2017

Phishing with Unicode Domains

“Phishing with Unicode Domains” is an interesting phishing attack resurfaced by a security researcher named Xudong Zheng last week.
In this phishing attack, it is impossible to identify whether a site is fraudulent one by just checking the SSL certificate + the URL in the URL bar.

Punycode makes it possible to register domains with foreign characters. It works by converting individual domain label to an alternative format using only ASCII characters. For example, the domain "xn--s7y.co" is equivalent to "短.co".
From a security perspective, Unicode domains can be problematic because many Unicode characters are difficult to distinguish from common ASCII characters. It is possible to register domains such as "xn--pple-43d.com", which is equivalent to "аpple.com". It may not be obvious at first glance, but "аpple.com" uses the Cyrillic "а" (U+0430) rather than the ASCII "a" (U+0041). This is known as a homograph attack.

Original article: https://www.xudongz.com/blog/2017/idn-phishing/
POC: https://www.xn--80ak6aa92e.com/
https://en.wikipedia.org/wiki/IDN_homograph_attack


~~~


Sunday, October 30, 2016

Oracle WebLogic Server - Part 2 - Node Manager and its interaction with WebLogic Server instances

In this post, I am going to describe node manager process and its interaction with WebLogic server instances. This post covers mostly theoretical parts, it does not includes instructions on  how to perform tasks related to node manager.

 In WebLogic Server environment, the Admin Server does not connect to the managed servers directly. A separate utility named "node manager" should be running in each machine for helping this purpose.

The main task of node manager is to provide a remote controller interface for controlling of the WebLogic Server instances (Admin Server and managed server instances) which are located in the same machine as the node manager process. It is not mandatory to have a node manager in a machine if you do not want to control  managed server instances through Admin Server. Also it is possible to have only one node manager for all the domains in a machine.

It is recommended to install node manager as an operating system service or daemon. Then the node manager will be automatically restarted when a system failure or reboot of the machine happens.

Node manager is required if you want following features,

Purpose of Node Manager Process:

  • Allow a remote or a local client to stop/start WebLogic server instances in the machine which runs the Node Manager process (mainly, but not necessarily from the Admin Server)
    • start, shutdown Admin Server instance
    • start, suspend, shutdown managed server instances
  • Automatically restart failed WebLogic server instances which were abnormally exited, if following conditions met,
    • the failed server instance was started by the node manager.
    • AutoRestart value of this server instance is set to true. (AutoRestart property of nodemanager's startup.properties file in <DOMAIN_HOME>/servers/<server_name>/data/nodemanager folder )
    • Number of max allowed restart is not yet passed. (RestartMax property of nodemanager's startup.properties file).
  • Allow monitoring of the health status of WebLogic servers from a remote or the local machine. (mainly, but not necessarily from the Admin Server)
  • Retrieve server logs and node manager logs.

Usually, the node manager gets the start/shutdown commands for starting/shut downing managed servers from the Admin Server. Then node manager is the process that actually start or shutdown the relevant managed server.

Tip: you can access the node manager (tested for a Plain Node Manager in WL 10.3.6) through a program like netcat, and then execute plain text commands to control the servers and monitor servers.
e.g:
~# nc <MACHINE_IP> 5556
domain <your_domain_name>
+OK Current domain set to ‘<your_domain_name>’
getStates
-ERR Must login to a domain first
user <weblogicuser>
+OK NM usr set to '<weblogicuser>'
pass <weblogicuser_password>
+OK Password accepted
getStates
+OK managed1=RUNNING managed2=UNKNOWN AdminServer=UNKNOWN
There are two versions of node manager processes.

Versions of Node Manager

Java based node manager

Java-based node manager process runs in a JVM instance. A Java-based node manager provides more security than a script based node manager.
  • Java-based node manager supports both Windows and UNIX operating systems.
  • Database-less leasing (consensus leasing) is faster when using a Java-based node manager.
  • Using inetd in UNIX, java-based node managers can be configured to restart automatically when receiving a request on a configured port.
  • Determines its configuration from nodemanager.properties file.
See https://docs.oracle.com/cd/E14571_01/web.1111/e13740/java_nodemgr.htm#NODEM177 for more details.

Script based node manager

You can managed remote server instances in a network through ssh using a script-based node manager (a script named wlscontrol.sh), using the scripts provided in this node manager version.
  • Script-based node manager only supports UNIX operating systems.
  • Automatic server migration is only supported by script-based node manager.
  • The node manager can control servers in remote machines through a set of scripts copied to those remote machine.
  • It requires only a smaller foot print than java-based node manager.
  • It supports SSH for security and setting up security is easier than setting up SSL for java based node manager.
  • It supports SSH or RSH for connection.
Following command line option should be added to Admin Server startup parameters in order to use SSH type of script based node manager.
-Dweblogic.nodemanager.ShellCommand='ssh -o PasswordAuthentication=no %H wlscontrol.sh -d %D -r %R -s %S %C'
where %H = host name of ssh server, %S = WebLogic server name, %D = WebLogic domain name, %R = domain directory, %C = node manager script command. For RSH type, the ShellCommand to use is relevant RSH command.

See http://docs.oracle.com/cd/E12840_01/wls/docs103/nodemgr/script_nodemgr.html  and "Running Script-based Node Manager" in https://docs.oracle.com/cd/E24329_01/web.1211/e21050/starting_nodemgr.htm#NODEM214 for more details about Script based node manager.

Node Manager Types

There are several node manager types which supports different level/type of security for node manager communication.
TypeDescriptionJava Based Node ManagerScript Based Node Manager
PlainCommunication is not encrypted.
SSLUse SSL for encrypting communication. You need to setup a certificate for this. See http://www.veriton.co.uk/roller/fmw/entry/ssl_for_weblogic_server_11g and http://weblogic-wonders.com/weblogic/2015/04/23/configure-ssl-nodemanager-12c/
SSHUse SSH session to do the communication. An OS user should be created for connecting using SSH in each machine and setup SSH settings for connection. See https://docs.oracle.com/cd/E23943_01/web.1111/e13740/script_nodemgr.htm#NODEM208
RSHThis type of node manager uses RSH for communication. But RSH is an older protocol which does not use encryption when sending data (such as passwords). Therefore RSH is not secure. It seems it is now rare to use this type of node manager, as not much information about it available in the internet.

Node Manager Operations

Starting Admin Server using Node Manager

  1. A node manager client authenticate to the Node Manager using node manager user name and password (e.g. nmConnect from WLST). Node Manager authenticates the given node manager user credentials using nm_password.properties file.
  2. Then Node Manager receives a command to start Admin Server from the node manager client (e.g. nmStart from WLST). This command provides the server name, domain name, server user credentials and any additional start-up properties.
  3. Node Manager determines the domain directory for the Admin Server from nodemanager.domains file and authenticates the given user credentials using a local file which contains the encrypted user credentials.
  4. Node Manager creates relevant directories (server_name/logs, server_name/security, server_name/data/nodemanager etc.) and obtains start-up properties for the Admin Server.
    • Then it saves encrypted user name, password etc in boot.properties file.
    • It saves the startup properties obtained from Admin Server configuration in startup.properties.
  5. Node Manager creates Admin Server process and start monitoring it.
  6. Admin Server process gets domain configuration data from its config folder.

Starting a managed server using Node Manager

This process can be done either from Admin Console, WLST console or another node manager client. Following steps mentions how to start the server from Admin console.
  1. A user issues a start command from Admin Console to start a managed server.
  2. Then Admin Server connects to the node manager which is located on the same machine as the targeted managed server, authenticate to the node manager.
  3. Then Admin Server sends the start command for starting the managed server to the node manager (with server name, username/password for connecting to server and other start-up properties).
  4. Node manager starts the managed server process.
  5. Managed server contacts the Admin Server to check for updates to its configuration information.
  6. If there is any update to the domain configuration, managed server updates its local cache of configuration from the update it got from the Admin Server.

Restart Admin Server using Node Manager

  1. Node Manager determines from the exit code of Admin Server process that it needs to restart the Admin Server.
  2. Node Manager obtains the user name and password value needed for starting the Admin server from the boot.properties file, and the start-up properties from <servername>/data/nodemanager/startup.properties file.
  3. Node Manager uses the above retrieved information and starts the Admin Server process.
  4. Admin Server reads its configuration data and starts up.

Restart a managed server using Node Manager

  1. Node Manager determines from the last known status of the managed server process (+ other node manager settings) that it needs to restart that managed server.
  2. Node Manager retrieves the user name and password value needed for starting the managed server from the boot.properties file, and the startup properties from <servername>/data/nodemanager/startup.properties file.
  3. Node Manager uses the above retrieved information and starts the managed Server process.
  4. Managed server contacts the Admin server to check for updates of the configuration data. If it get any updates, it update its local cache of config directory with the received updates.
  5. If managed server fails to contact the Admin Server and if the Managed Server Independence Mode (MSI) is enabled, the managed server uses its locally cached configuration data.

Shutdown a server instance using Node Manager

  1. An authorized user connects to the Admin console and issues the shutdown command for a managed server.
  2. Then Admin Server issues shutdown command directly to the managed server. Then managed server perform the shutdown sequences for Graceful Shutdown procedure. 
  3. If Admin server failed to contact the managed server in above step, then it connects to the node manager and issues the shutdown command to the node manager for shutting down the managed server.
  4. Node manager asks Operating System to kill the managed server process.
  5. Operating system kills the managed server process.

Important files and folders related to node manager

FileFolderPurpose
nodemanager.properties<NODEMANAGER_HOME>/The configuration file used by Java based node manager
nodemanager.domains<NODEMANAGER_HOME>/Contains domain names and corresponding directories of the domains managed by the node manager
nm_data.properties<NODEMANAGER_HOME>/This file stores a symmetric encryption key. It is stored in encrypted form.
nm_password.properties<DOMAIN_HOME>/config/nodemanagerThis file stores the username and password of the node manager
boot.properties<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis files contains the user credentials needed for starting server instances.
startup.properties<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerEach managed server instance contains its own copy of startup.properties file which contains the properties that control the startup/shutdown/restart of the server instance by the node manager.
<server_name>.addr<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis file contains the IP address added when the server started or migrated.
<server_name>.pid<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis file contains the process id of the server instance. Node manager checks this process id during crash recovery. This is generated by the server instance.
<server_name>.state<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis file contains the current state of the server instance. This file is generated by the server instance and monitored by the node manager. Without this file, node manager cannot determine the current state of the server instance.
<server_name>.lck<DOMAIN_HOME>/servers/<server_name>/data/nodemanagerThis file is generated by the server instance and contains a internal lock id.
nodemanager.log<NODEMANAGER_HOME>/Node manager log file (only for Java based node manager). This file contains logs about all of domains controlled by the node manager. This log can be viewed using nmLog command of WLST and "Machines --> Monitoring --> Node Manager Log" in Admin Console.
<server_name>.out<DOMAIN_HOME>/servers/>server_name</logsNode Manager Server instance log file. This file contains the messages written to stderr and stdout by the server instance. You can view this log by either nmServerLog WLST command or "Diagnostics --> Log Files".
<server_name>.log<DOMAIN_HOME>/servers/<server_name>/logsWebLogic server log files.

References:



~~~


Friday, October 21, 2016

Class Loading in Java - Concept, Class Loader Types, and Related Exceptions

The other day one of my colleagues asked me whether I could explain to him about Java class loading. Then I thought "I know basics about Java Class Loading, but is it complete and accurate?". So I decided to do some search about the topic and it ended with writing this post.

When you launch a Java application using "java" command, one of major operation that happens behind it is the loading of required classes. There are 3 types of classes normally needed for running a Java application.
  1. Classes that defines the implementation of Java platform. These classes are necessary for running basic Java Runtime Environment (JRE).
  2. Classes that implements the functionalities for Java Extension mechanism.
  3. Classes that implements the functionality of your application and the 3rd party libraries required for running your application code.
Process of finding the byte code for a given class name and then converting that byte code to a JVM internal representation of a class, is known as class loading. JVM has 3 separate class loaders for loading each of above mentioned sets of classes. In addition to those, developers can create their own class loaders.
  1. Bootstrap class loader
  2. Extension class loader
  3. System classpath class loader
More details about these class loaders are explained in below sections. Few important things to remember is, 
  • Each of those class loaders has different set of classpaths (directories and jar files to find the classes) to check when it tries to find/load a class. (e.g.: Bootstrap class loader loads class located in "sun.boot.class.path" system property which normally point to a set of jar files in jre/lib folder.)
  • JVM does not load all classes at the start-up, it only loads the required set of classes, and then load the classes dynamically as needed. 
  • The same class loaded by two separate class loaders are not equal. Therefore you cannot cast one of those class to other class, although those were loaded from same class byte code. A loaded class is internally identified by [the package name, class name, the name of the class loader which it loaded]. (Two different class loaders mentioned here does not have parent-child relationship. if that is the case, the class is loaded by same parent.)
The classes loaded by Class Loaders are stored in method area in PermGen area of JVM. Method Area stores the per-class information such as 
  • Class Loader Reference
  • Runtime Constant Pool
  • Field Data
  • Method Data
  • Method Code (Byte code + other information)
All threads share the same Method Area.

Java has following class loader types in addition to user built class loaders.

Class Loader Types

Bootstrap Class Loader (aka Primordial or Null Class Loader)

    Bootstrap Class Loader is the parent class loader of all other class loaders. It is responsible for loading the key classes (from java.* packages etc.) at the beginning of the class loading. It loads the pre-requisite classes needed for supporting a basic Java Runtime Environment. It has access to a set of trusted classes which can be run by JVM without "verification". Bootstrap Class Loader is implemented using native code (mostly in C), not using Java code. But all other Class Loaders are implemented using Java code.
    So when you execute a java program with -verbose:class JVM argument, You can see that first few lines in the output that starts with "[Loaded" are related to the class loading done by Bootstrap Class Loader from jre\lib\rt.jar file and other important jar files the jre\lib directory in your Java installation. This location is known as bootstrap classpath and you can read it from sun.boot.class.path system property.
    You can use -Xbootclasspath non-standard command line option to change the directories of bootstrap class path, but it is not recommended and it violates JRE binary code license. There are two more variation of this command line option; -Xbootclasspath/a  to append to existing bootclasspath, and -Xbootclasspath/p to prepend to existing bootclasspath.

Extension Class Loader

Extension Class Loader is immediate child of the Bootstrap Class Loader. Extension Class Loader is responsible for loading classes from Java extensions directories (value of java.ext.dirs system property including "jre\lib\ext"). These locations has the jar files which implements the Java extension APIs.
Extension Class Loader in JVM is implemented by sun.misc.Launcher$ExtClassLoader. When Extension Class Loader get a call to load a class, it first tries to delegate the loading to the Bootstrap class loader (because it is the parent class loader). If Bootstrap class loader cannot load the class, then Extension Class Loader tries to load the class from "jre\lib\ext" directory or other directories listed in "java.ext.dirs" system property.

System Class Loader (System Classpath Class Loader)

System Class Loader is the immediate child of the Extension Class Loader. This class loader loads the classes from classes and jar files in CLASSPATH environment variable or given in -classpath command line option.
Some important points about Classpath of the System Class Loader,
  • Default classpath is "." aka current directory.
  • You can use CLASSPATH environment variable to define the classpath.
  • You can use -cp or -classpath command line option to define the classpath, this will override the value defined in CLASSPATH environment variable.
  • When you use -jar command line option, it will ignore the values in -cp and -classpath command line options, and also the value of CLASSPATH environment variable. It will only consider the Class-Path attribute defined in the META-INF/mainfest.mf file of the jar file.
  • If two classes with the same name exists in the classpaths, the class loader will load the first found class.
  • Current value of the classpath being used by a java program can be retrieved from java.class.path system property.
System Class Loader is implemented by sun.misc.Launcher$AppClassLoader class and it is a child of Extension Class Loader. You can change the System Classpath Class Loader to use your own class loader implementation instead of sun.misc.Launcher$AppClassLoader, by using -Djava.system.class.loader command line option.

User Defined Class Loaders

Developers can create their own class loaders to support specific requirements of their application. Possible examples for implementing user defined class loaders,
  • to load classes through network.
  • to load classes which are auto generated at runtime.
  • to load classes from encrypted files.
  • to separate class loading of different applications which runs in same JVM.
  • etc.

Class Loading Algorithm

Class loaders (at least JVM class loaders) follow following algorithm when it wants to load a class. (This algorithm is based on delegating the task to parent first.)
  1. Check whether class is already available in the local cache of the current class loader.
  2. If the class was not found in the cache, ask (delegate) the parent class loader to load the class. (Please remember that Bootstrap Class Loader does not have a parent.)
  3. Parent class loader also does above steps recursively until the class found in its cache or no parent class loader found. (Ask parent's parent class loader to load, if parent class loader did not find the class in its cache.).
  4. If the parent class loader can find and load the class, it store it in its cache and return it to its child class loader.
  5. If parent class loader cannot load the class, the class loader tries to load the class from its classpaths (aka code sources).
  6. If the class loader cannot find the class in its local cache and cannot get the class from parent class loaders and cannot load the class from its local code sources, it throws ClassNotFoundException.
Example: When system classpath class loader wants to load javax.imageio.ImageIO class (a class belong to Java Extensions),
  1. System Classpath Class Loader check whether it has javax.imageio.ImageIO class loaded in its local cache.
  2. It does not find it in local cache, so it ask its parent class loader (which is Extensions Class Loader) to load the javax.imageio.ImageIO class.
  3. Extension Class Loader checks its local cache whether it has javax.imageio.ImageIO class.
  4. If it found the javax.imageio.ImageIO class in its local cache, it returns it to the System Classpath Class Loader.
  5. If it did not find the javax.imageio.ImageIO class in its local cache, it ask its parent class loader (which is Bootstrap Class Loader) to load the javax.imageio.ImageIO class.
  6. Bootstrap Class Loader checks whether it has javax.imageio.ImageIO in its local cache. If it has the class in local cache, it returns it to Extension Class Loader method call. (But in our example, Bootstrap class loader should not find the class in its local cache, because the class is in extension classpath).
  7. If Bootstrap Class Loader did not find it in its local cache, and because it does not have any parent class loader, it tries to load the javax.imageio.ImageIO class from its classpath aka its local code sources (bootstrap classpath).
  8. Because the class is belong to Java Extensions, Bootstrap Class Loader does not find it in bootstrap classpath. Then it return the method call to its child class loader (Extension Class Loader) without any class loaded.
  9. Then Extension Class Loader tries to find the javax.imageio.ImageIO class in its classpath aka its local code sources.
  10. Because the class belongs to Java Extensions, Extension Class Loader finds the class in its classpath and loads it. Then put the class in its local cache. Then it returns the class to its child class loader which is System Classpath Class Loader.
    So when a class loader wants to load a class, it first check whether class is already loaded in its local cache, then delegates the loading to the parent class loader if it is not in the current class loader's cache. The class loader will try to load the class itself, only if the parent class loader cannot load the class.
    Therefore the classes loaded by parent class loaders are visible to the children class loaders of that parent class loader. But the classes loaded by a child class loader are not visible to any of parent class loader of the child. Due to this algorithm, a class loaded by parent class loader will not be reloaded by a child class loader of it.


Stages in Class Loading

There are 3 main stages in loading a class from byte code to proper JVM internal representation.
  1. Loading - Class Loader search for the class file (byte code) in the classpath and the byte code is loaded to memory (as a ClassFile structure). This step creates a basic structure for the class, but it does not have all the important data filled yet.
  2. Linking - This step tries to convert byte code data to proper runtime information. This step is sub divided into 3 more sub steps.
    1. Byte Code Verification - the byte code is checked for the correct format and other checks. This step ensures that loaded class is structurally correct.
    2. Class Preparation - This step prepare the data structures to represent methods, fields, and implemented interfaces etc. from the byte code. This step create static fields and initialize those to their default values.
    3. Resolving - This step do the resolving of other classes referenced by this class. (e.g.: Super classes, field types, method param types, method local variable types)
  3. Initializing - This step executes the static initializer blocks.

Class Data Sharing (CDS) feature

 Class Data Sharing (CDS) is a feature introduced in JavaSE 5.0 (in Oracle/Sun hotspot JVM) in order to reduce the java application start-up time by improving class loading time. When the JRE is installed in your machine, JRE loads the classes from system jar files into internal representations and then dump those loaded internal representations to a file known as "shared archive" (which is located in jre/lib/[arch]/client/classes.jsa in UNIX and in jre/bin/client/classes.jsa in Windows). During subsequent JVM invocations, the "shared archive" is used by those JVM processes by memory mapping to it, so reducing the loading time of classes loaded by Bootstrap class loader (which also means this memory mapped file is shared among multiple JVM processes).

If this feature is enabled, you will see many lines with "[Loaded className from shared objects file]" when you run a java application with "-verbose:class" command line argument. The list of classes in the "shared archive" can be found in "jre/lib/classlist" file.

You can use "-XShare" command line option to either enable or disable class data sharing for your java application. 

In JDK 8U40, Oracle has introduced an experimental feature named Application Class Data Sharing (AppCDS) by extending CDS, to allow developers to place classes from Standard Extensions directories and from Application classpath in the "shared archive".

Common Errors/Exceptions related to class loading

ClassNotFoundException

java.lang.ClassNotFoundException is thrown when your program dynamically tries to load the class at runtime and the class loader did not find the requested class. Main reason is, the class is not available in the CLASSPATH.
This is normally thrown by methods like Class.forName(), ClassLoader.loadClass(), ClassLoader.findSystemClass() when the java program tries to load a class dynamically, using its string name. Due to this reason, the compiler does not detect the unavailability of the class at the compile time.

NoClassDefFoundError

java.lang.NoClassDefFoundError happens when JVM or a class loader instance trying to load a class definition, but this class definition cannot be found or loaded due to some reason.
Most of the time, the class was available in the classpath during the compile time, but it is no longer available during the runtime. Another possible reason is, the class is still available in the classpath, but the class has static initialization blocks which fails during the loading of the class.

References:



~~~