HDP upgrade

After upgrade Ambari to lastest version then

  1. Backup configure files /etc/hadoop/conf /etc/hive-hcatalog/conf /etc/hive-webhcat/conf /etc/hive/conf /etc/pig/conf /etc/sqoop/conf /etc/mahout/conf /etc/oozie/conf /etc/hue/conf /etc/zookeeper/conf /etc/tez/conf

  2. Stop all services (including MapReduce) except HDFS, ZooKeeper, and Ranger, and client
    applications deployed on HDFS.

  3. Back up the Hive Metastore database. Postgres Backup: sudo -u $username pg_dump $databasename > $outputfilename.sql sbr For example: sudo -u postgres pg_dump hive > /tmp/mydir/backup_hive.sql

    Restore: sudo -u $username psql $databasename < $inputfilename.sqlsbr For example: sudo -u postgres psql hive < /tmp/mydir/backup_hive.sql

  4. Save the namespace by executing the following commands: su - hdfs hdfs dfsadmin -safemode enter hdfs dfsadmin -saveNamespace

  5. Run the fsck command as the HDFS Service user and fix any errors. su - hdfs -c "hdfs fsck / -files -blocks -locations > dfs-old-fsck-1.log"

  6. Use the following instructions to compare status before and after the upgrade.

    • Capture the complete namespace of the file system. su - hdfs -c "hdfs dfs -ls -R / > dfs-old-lsr-1.log"
    • Run the report command to create a list of DataNodes in the cluster. su - hdfs dfsadmin -c "-report > dfs-old-report-1.log"
    • Run the report command to create a list of DataNodes in the cluster. su - hdfs -c "hdfs dfsadmin -report > dfs-old-report-l.log"
  1. Stop remaining services (HDFS, Zookeeper, and Ranger).

  2. Back up your NameNode metadata.

    • Copy the following checkpoint files into a backup directory. The NameNode metadata is stored in a directory specified in the hdfs-site.xml configuration file
      under the configuration value "dfs.namenode.dir".
    • Store the layoutVersion of the namenode. ${dfs.namenode.name.dir}/current/VERSION

  3. Verify that edit logs in ${dfs.namenode.name.dir}/current/edits* are empty.

    • Run: hdfs oev -i ${dfs.namenode.name.dir}/current/editsinprogress* -o edits.out
    • Verify the edits.out file. It should only have OP_START_LOG_SEGMENT transaction. <?xml version="1.0" encoding="UTF-8"?> -56 OP_START_LOG_SEGMENT 5749
    • If edits.out has transactions other than OP_START_LOG_SEGMENT, run the following steps and then verify edit logs are empty. Start the existing version NameNode. Ensure there is a new FS image file. Shut the NameNode down: hdfs dfsadmin – saveNamespace

  4. Upgrade HDP 2.3 Components (on all hosts)

    • yum clean all
    • yum erase "accumulo$VERSION$BUILD" "atlasmetadata$VERSION_$BUILD" "datafu$VERSION$BUILD" "falcon$VERSION$BUILD" "flume$VERSION$BUILD" "hadoop$VERSION$BUILD"
      "hadoop$VERSION$BUILD-client" "hadoop$VERSION$BUILD-hdfs" "hadooplzo$VERSION$BUILD*" ...
    • Validate that all HDP 2.3 component binaries are uninstalled: yum list installed | grep @HDP2.3
    • Remove your old hdp.repo file: rm /etc/yum.repos.d/hdp.repo
    • Install the HDP 2.4.X.X repo wget -nv http://public-repo-1.hortonworks.com/HDP/centos6/2.x/updates/2.4.x.x/hdp.repo -O
      /etc/yum.repos.d/hdp.repo yum repolist
    • yum install "accumulo$VERSION$BUILD" "datafu$VERSION$BUILD" "falcon$VERSION$BUILD" "flume$VERSION$BUILD" "hadoop$VERSION$BUILD" "hadooplzo$VERSION$BUILD" ...
    • yum list installed | grep HDP-

  5. (Warning) HDP 2.4.0 installs hdp-select automatically with the installation or upgrade of the
    first HDP component. If you have not already upgraded ZooKeeper, hdp-select has not been installed.     To prevent version-specific directory issues for your scripts and updates, Hortonworks provides hdp- select, a script that symlinks directories to hdp-current and modifies paths for configuration
    directories.

    • rm /usr/bin/oozie
    • hdp-select set all 2.4.x.x-$BUILD

  6. Configure and Start Apache ZooKeeper

  7. Restart Ambari-server
  8. Restart all-service

Reference:

  1. https://docs.hortonworks.com/HDPDocuments/HDP2/HDP-
    2.4.2/bk_upgrading_hdp_manually/bk_upgrading_hdp_manually-20160509.pdf

  2. http://docs.hortonworks.com/HDPDocuments/Ambari- 2.2.2.0/bk_upgrading_Ambari/content/_upgrading_hdp_stack.html

results matching ""

    No results matching ""