1. Introduction
  2. Prerequistes
  3. Set Environment Variables
  4. Setup SSH daemon
  5. Download hadoop and place it in the home directory.
  6. Unpack hadoop
  7. Configure Hadoop
  8. Format the namenode
  9. Setup hadoop plugin
  10. Start the cluster
  11. Setup hadoop location
  12. Upload data
  13. Create and run a test project.
Bookmark and Share

Setup SSH daemon

Both Hadoop scripts and Eclipse plug-in need password-less SSH to operate. This section describes how to set it up in the Cygwin environment.

Configure ssh daemon

  1. Open the Cygwin command prompt.
  2. Execute the following command:
  3. When asked if privilege separation should be used, answer no.
  4. When asked if sshd should be installed as a service, answer yes.
  5. When asked about the value of CYGWIN environment variable, enter ntsec.
  6. Here is an example session of this command. Note that the input typed by the user is shown in pink and output from the system is shown in gray.

    Example of using ssh-host-config

Start SSH daemon

  1. Find My Computer icon either on your desktop or in the start-up menu, right-click on it and select Manage from the context menu.
  2. Open Services and Applications in the left-hand panel then select the Services item.
  3. Find the CYGWIN sshd item in the main section and right-click on it.
  4. Select Start from the context menu.

Start SSHD service

  • A small window should pop-up indicating the progress of the service start-up. After that window disappears the status of CYGWIN sshd service should change to Started.
  • Setup authorization keys

    Eclipse plug-in and Hadoop scripts require ssh authentication to be performed through authorization keys rather than passwords. The following steps describe how authorization keys are set up.

    1. Open cygwin command prompt
    2. Execute the following command to generate keys
    3. When prompted for filenames and pass phrases press ENTER to accept default values.
    4. After the command has finished generating keys, enter the following command to change into your .ssh directory:
      cd ~/.ssh
    5. Check if the keys were indeed generated by executing the following command:
      ls -l

      You should see two files id_rsa.pub and id_rsa with recent creation dates. These files contain authorization keys.
    6. To register the new authorization keys enter the following command (note the sharply-angled double brackets -- they are very important):
      cat id_rsa.pub >> authorized_keys

      Setting up authorization keys

    7. Now check if the keys were set up correctly by executing the following command:
      ssh localhost
      Since it is a new ssh installation, you will be warned that authenticity of the host could not be established and will be asked whether you really want to connect. Answer yes and press ENTER. You should see the Cygwin prompt again, which means that you have successfully connected.
    8. Now execute the command again:
      ssh localhost
      This time you should not be prompted for anything.



    Bookmark and Share