|
|
按照新的文档来 更新配置: http://hadoop.apache.org/common/docs/current/cluster_setup.html
发现多了很多东西,新的文档也比以前的详细,好的多.
不过此次只为了升级hadoop0.20.2到hadoop-0.21.0
看看发布的版本:http://hadoop.apache.org/common/releases.html
23
August, 2010: release 0.21.0 available
26
February, 2010: release 0.20.2 available
整整半年了,等的好辛苦,等得Hbase也很辛苦。 本次版本的更新，修复了N多的BUG ，看看吧：
http://hadoop.apache.org/common/docs/r0.21.0/changes.html
着实吓了一跳：
INCOMPATIBLE CHANGES
(31)
HADOOP-4895
.
Remove deprecated methods DFSClient.getHints(..) and
DFSClient.isDirectory(..).
(szetszwo)
HADOOP-4941
.
Remove deprecated FileSystem methods: getBlockSize(Path f),
getLength(Path f) and getReplication(Path src).
(szetszwo)
HADOOP-4648
.
Remove obsolete, deprecated InMemoryFileSystem and
ChecksumDistributedFileSystem.
(cdouglas via szetszwo)
HADOOP-4940
.
Remove a deprecated method FileSystem.delete(Path f).
(Enis
Soztutar via szetszwo)
HADOOP-4010
.
Change semantics for LineRecordReader to read an additional
line per split- rather than moving back one character in the stream- to
work with splittable compression codecs.
(Abdul Qadeer via cdouglas)
HADOOP-5094
.
Show hostname and separate live/dead datanodes in DFSAdmin
report.
(Jakob Homan via szetszwo)
HADOOP-4942
.
Remove deprecated FileSystem methods getName() and
getNamed(String name, Configuration conf).
(Jakob Homan via szetszwo)
HADOOP-5486
.
Removes the CLASSPATH string from the command line and instead
exports it in the environment.
(Amareshwari Sriramadasu via ddas)
HADOOP-2827
.
Remove deprecated NetUtils::getServerAddress.
(cdouglas)
HADOOP-5681
.
Change examples RandomWriter and RandomTextWriter to
use new mapreduce API.
(Amareshwari Sriramadasu via sharad)
HADOOP-5680
.
Change org.apache.hadoop.examples.SleepJob to use new
mapreduce api.
(Amareshwari Sriramadasu via sharad)
HADOOP-5699
.
Change org.apache.hadoop.examples.PiEstimator to use
new mapreduce api.
(Amareshwari Sriramadasu via sharad)
HADOOP-5720
.
Introduces new task types - JOB_SETUP, JOB_CLEANUP
and TASK_CLEANUP. Removes the isMap methods from TaskID/TaskAttemptID
classes.
(ddas)
HADOOP-5668
.
Change TotalOrderPartitioner to use new API.
(Amareshwari
Sriramadasu via cdouglas)
HADOOP-5738
.
Split "waiting_tasks" JobTracker metric into waiting maps and
waiting reduces.
(Sreekanth Ramakrishnan via cdouglas)
HADOOP-5679
.
Resolve findbugs warnings in core/streaming/pipes/examples.
(Jothi
Padmanabhan via sharad)
HADOOP-4359
.
Support for data access authorization checking on Datanodes.
(Kan
Zhang via rangadi)
HADOOP-5690
.
Change org.apache.hadoop.examples.DBCountPageView to use
new mapreduce api.
(Amareshwari Sriramadasu via sharad)
HADOOP-5694
.
Change org.apache.hadoop.examples.dancing to use new
mapreduce api.
(Amareshwari Sriramadasu via sharad)
HADOOP-5696
.
Change org.apache.hadoop.examples.Sort to use new
mapreduce api.
(Amareshwari Sriramadasu via sharad)
HADOOP-5698
.
Change org.apache.hadoop.examples.MultiFileWordCount to
use new mapreduce api.
(Amareshwari Sriramadasu via sharad)
HADOOP-5913
.
Provide ability to an administrator to stop and start
job queues.
(Rahul Kumar Singh and Hemanth Yamijala via yhemanth)
- MAPREDUCE-711. Removed Distributed Cache from Common, to move
it
under Map/Reduce.
(Vinod Kumar Vavilapalli via yhemanth)
HADOOP-6201
.
Change FileSystem::listStatus contract to throw
FileNotFoundException if the directory does not exist, rather than
letting
this be implementation-specific.
(Jakob Homan via cdouglas)
HADOOP-6230
.
Moved process tree and memory calculator related classes
from Common to Map/Reduce.
(Vinod Kumar Vavilapalli via yhemanth)
HADOOP-6203
.
FsShell rm/rmr error message indicates exceeding Trash quota
and suggests using -skpTrash, when moving to trash fails.
(Boris
Shkolnik via suresh)
HADOOP-6303
.
Eclipse .classpath template has outdated jar files and is
missing some new ones.
(cos)
HADOOP-6396
.
Fix uninformative exception message when unable to parse
umask.
(jghoman)
HADOOP-6299
.
Reimplement the UserGroupInformation to use the OS
specific and Kerberos JAAS login.
(omalley)
HADOOP-6686
.
Remove redundant exception class name from the exception
message for the exceptions thrown at RPC client.
(suresh)
HADOOP-6701
.
Fix incorrect exit codes returned from chmod, chown and chgrp
commands from FsShell.
(Ravi Phulari via suresh)
NEW FEATURES
(59)
HADOOP-6332
.
Large-scale Automated Test Framework.
(sharad, Sreekanth
Ramakrishnan, at all via cos)
HADOOP-4268
.
Change fsck to use ClientProtocol methods so that the
corresponding permission requirement for running the ClientProtocol
methods will be enforced.
(szetszwo)
HADOOP-3953
.
Implement sticky bit for directories in HDFS.
(Jakob Homan
via szetszwo)
HADOOP-4368
.
Implement df in FsShell to show the status of a FileSystem.
(Craig
Macdonald via szetszwo)
HADOOP-3741
.
Add a web ui to the SecondaryNameNode for showing its status.
(szetszwo)
HADOOP-5018
.
Add pipelined writers to Chukwa.
(Ari Rabkin via cdouglas)
HADOOP-5052
.
Add an example computing exact digits of pi using the
Bailey-Borwein-Plouffe algorithm. (Tsz Wo (Nicholas), SZE via cdouglas)
HADOOP-4927
.
Adds a generic wrapper around outputformat to allow creation of
output on demand
(Jothi Padmanabhan via ddas)
HADOOP-5144
.
Add a new DFSAdmin command for changing the setting of restore
failed storage replicas in namenode.
(Boris Shkolnik via szetszwo)
HADOOP-5258
.
Add a new DFSAdmin command to print a tree of the rack and
datanode topology as seen by the namenode.
(Jakob Homan via szetszwo)
HADOOP-4756
.
A command line tool to access JMX properties on NameNode
and DataNode.
(Boris Shkolnik via rangadi)
HADOOP-4539
.
Introduce backup node and checkpoint node.
(shv)
HADOOP-5363
.
Add support for proxying connections to multiple clusters with
different versions to hdfsproxy.
(Zhiyong Zhang via cdouglas)
HADOOP-5528
.
Add a configurable hash partitioner operating on ranges of
BinaryComparable keys.
(Klaas Bosteels via shv)
HADOOP-5257
.
HDFS servers may start and stop external components through
a plugin interface.
(Carlos Valiente via dhruba)
HADOOP-5450
.
Add application-specific data types to streaming's typed bytes
interface.
(Klaas Bosteels via omalley)
HADOOP-5518
.
Add contrib/mrunit, a MapReduce unit test framework.
(Aaron Kimball
via cutting)
HADOOP-5469
.
Add /metrics servlet to daemons, providing metrics
over HTTP as either text or JSON.
(Philip Zeyliger via cutting)
HADOOP-5467
.
Introduce offline fsimage image viewer.
(Jakob Homan via shv)
HADOOP-5752
.
Add a new hdfs image processor, Delimited, to oiv.
(Jakob
Homan via szetszwo)
HADOOP-5266
.
Adds the capability to do mark/reset of the reduce values
iterator in the Context object API.
(Jothi Padmanabhan via ddas)
HADOOP-5745
.
Allow setting the default value of maxRunningJobs for all
pools.
(dhruba via matei)
HADOOP-5643
.
Adds a way to decommission TaskTrackers while the JobTracker
is running.
(Amar Kamat via ddas)
HADOOP-4829
.
Allow FileSystem shutdown hook to be disabled.
(Todd Lipcon via
tomwhite)
HADOOP-5815
.
Sqoop: A database import tool for Hadoop.
(Aaron Kimball via
tomwhite)
HADOOP-4861
.
Add disk usage with human-readable size (-duh).
(Todd Lipcon via
tomwhite)
HADOOP-5844
.
Use mysqldump when connecting to local mysql instance in Sqoop.
(Aaron
Kimball via tomwhite)
HADOOP-5976
.
Add a new command, classpath, to the hadoop script.
(Owen
O'Malley and Gary Murry via szetszwo)
HADOOP-6120
.
Add support for Avro specific and reflect data.
(sharad via cutting)
HADOOP-6226
.
Moves BoundedByteArrayOutputStream from the tfile package to
the io package and makes it available to other users (MAPREDUCE-318).
(Jothi
Padmanabhan via ddas)
HADOOP-6105
.
Adds support for automatically handling deprecation of
configuration keys.
(V.V.Chaitanya Krishna via yhemanth)
HADOOP-6235
.
Adds new method to FileSystem for clients to get server
defaults.
(Kan Zhang via suresh)
HADOOP-6234
.
Add new option dfs.umaskmode to set umask in configuration
to use octal or symbolic instead of decimal.
(Jakob Homan via suresh)
HADOOP-5073
.
Add annotation mechanism for interface classification.
(Jakob Homan
via suresh)
HADOOP-4012
.
Provide splitting support for bzip2 compressed files.
(Abdul
Qadeer via cdouglas)
HADOOP-6246
.
Add backward compatibility support to use deprecated decimal
umask from old configuration.
(Jakob Homan via suresh)
HADOOP-4952
.
Add new improved file system interface FileContext for the
application writer
(Sanjay Radia via suresh)
HADOOP-6170
.
Add facility to tunnel Avro RPCs through Hadoop RPCs.
This permits one to take advantage of both Avro's RPC versioning
features and Hadoop's proven RPC scalability.
(cutting)
HADOOP-6267
.
Permit building contrib modules located in external
source trees.
(Todd Lipcon via cutting)
HADOOP-6240
.
Add new FileContext rename operation that posix compliant
that allows overwriting existing destination.
(suresh)
HADOOP-6204
.
Implementing aspects development and fault injeciton
framework for Hadoop
(cos)
HADOOP-6313
.
Implement Syncable interface in FSDataOutputStream to expose
flush APIs to application users.
(Hairong Kuang via suresh)
HADOOP-6284
.
Add a new parameter, HADOOP_JAVA_PLATFORM_OPTS, to
hadoop-config.sh so that it allows setting java command options for
JAVA_PLATFORM.
(Koji Noguchi via szetszwo)
HADOOP-6337
.
Updates FilterInitializer class to be more visible,
and the init of the class is made to take a Configuration argument.
(Jakob
Homan via ddas)
- Hadoop-6223. Add new file system interface AbstractFileSystem
with
implementation of some file systems that delegate to old FileSystem.
(Sanjay
Radia via suresh)
HADOOP-6433
.
Introduce asychronous deletion of files via a pool of
threads. This can be used to delete files in the Distributed
Cache.
(Zheng Shao via dhruba)
HADOOP-6415
.
Adds a common token interface for both job token and
delegation token.
(Kan Zhang via ddas)
HADOOP-6408
.
Add a /conf servlet to dump running configuration.
(Todd Lipcon via
tomwhite)
HADOOP-6520
.
Adds APIs to read/write Token and secret keys. Also
adds the automatic loading of tokens into UserGroupInformation
upon login. The tokens are read from a file specified in the
environment variable.
(ddas)
HADOOP-6419
.
Adds SASL based authentication to RPC.
(Kan Zhang via ddas)
HADOOP-6510
.
Adds a way for superusers to impersonate other users
in a secure environment.
(Jitendra Nath Pandey via ddas)
HADOOP-6421
.
Adds Symbolic links to FileContext, AbstractFileSystem.
It also adds a limited implementation for the local file system
(RawLocalFs) that allows local symlinks.
(Eli Collins via Sanjay
Radia)
HADOOP-6577
.
Add hidden configuration option "ipc.server.max.response.size"
to change the default 1 MB, the maximum size when large IPC handler
response buffer is reset.
(suresh)
HADOOP-6568
.
Adds authorization for the default servlets.
(Vinod Kumar
Vavilapalli via ddas)
HADOOP-6586
.
Log authentication and authorization failures and successes
for RPC
(boryas)
HADOOP-6580
.
UGI should contain authentication method.
(jnp via boryas)
HADOOP-6657
.
Add a capitalization method to StringUtils for MAPREDUCE-1545.
(Luke
Lu via Steve Loughran)
HADOOP-6692
.
Add FileContext#listStatus that returns an iterator.
(hairong)
HADOOP-6869
.
Functionality to create file or folder on a remote daemon
side
(Vinay Thota via cos)
IMPROVEMENTS
(198)
HADOOP-6798
.
Align Ivy version for all Hadoop subprojects.
(cos)
HADOOP-6777
.
Implement a functionality for suspend and resume a process.
(Vinay
Thota via cos)
HADOOP-6772
.
Utilities for system tests specific.
(Vinay Thota via cos)
HADOOP-6771
.
Herriot's artifact id for Maven deployment should be set to
hadoop-core-instrumented
(cos)
HADOOP-6752
.
Remote cluster control functionality needs JavaDocs
improvement (Balaji Rajagopalan via cos).
HADOOP-4565
.
Added CombineFileInputFormat to use data locality information
to create splits.
(dhruba via zshao)
HADOOP-4936
.
Improvements to TestSafeMode.
(shv)
HADOOP-4985
.
Remove unnecessary "throw IOException" declarations in
FSDirectory related methods.
(szetszwo)
HADOOP-5017
.
Change NameNode.namesystem declaration to private.
(szetszwo)
HADOOP-4794
.
Add branch information from the source version control into
the version information that is compiled into Hadoop.
(cdouglas via
omalley)
HADOOP-5070
.
Increment copyright year to 2009, remove assertions of ASF
copyright to licensed files. (Tsz Wo (Nicholas), SZE via cdouglas)
HADOOP-5037
.
Deprecate static FSNamesystem.getFSNamesystem().
(szetszwo)
HADOOP-5088
.
Include releaseaudit target as part of developer test-patch
target.
(Giridharan Kesavan via nigel)
HADOOP-2721
.
Uses setsid when creating new tasks so that subprocesses of
this process will be within this new session (and this process will be
the process leader for all the subprocesses). Killing the process
leader,
or the main Java task in Hadoop's case, kills the entire subtree of
processes.
(Ravi Gummadi via ddas)
HADOOP-5097
.
Remove static variable JspHelper.fsn, a static reference to
a non-singleton FSNamesystem object.
(szetszwo)
HADOOP-3327
.
Improves handling of READ_TIMEOUT during map output copying.
(Amareshwari
Sriramadasu via ddas)
HADOOP-5124
.
Choose datanodes randomly instead of starting from the first
datanode for providing fairness.
(hairong via szetszwo)
HADOOP-4930
.
Implement a Linux native executable that can be used to
launch tasks as users.
(Sreekanth Ramakrishnan via yhemanth)
HADOOP-5122
.
Fix format of fs.default.name value in libhdfs test conf.
(Craig
Macdonald via tomwhite)
HADOOP-5038
.
Direct daemon trace to debug log instead of stdout.
(Jerome
Boulon via cdouglas)
HADOOP-5101
.
Improve packaging by adding 'all-jars' target building core,
tools, and example jars. Let findbugs depend on this rather than the
'tar'
target.
(Giridharan Kesavan via cdouglas)
HADOOP-4868
.
Splits the hadoop script into three parts - bin/hadoop,
bin/mapred and bin/hdfs.
(Sharad Agarwal via ddas)
HADOOP-1722
.
Adds support for TypedBytes and RawBytes in Streaming.
(Klaas
Bosteels via ddas)
HADOOP-4220
.
Changes the JobTracker restart tests so that they take much
less time.
(Amar Kamat via ddas)
HADOOP-4885
.
Try to restore failed name-node storage directories at
checkpoint time.
(Boris Shkolnik via shv)
HADOOP-5209
.
Update year to 2009 for javadoc.
(szetszwo)
HADOOP-5279
.
Remove unnecessary targets from test-patch.sh.
(Giridharan Kesavan
via nigel)
HADOOP-5120
.
Remove the use of FSNamesystem.getFSNamesystem() from
UpgradeManagerNamenode and UpgradeObjectNamenode.
(szetszwo)
HADOOP-5222
.
Add offset to datanode clienttrace.
(Lei Xu via cdouglas)
HADOOP-5240
.
Skip re-building javadoc when it is already
up-to-date.
(Aaron Kimball via cutting)
HADOOP-5042
.
Add a cleanup stage to log rollover in Chukwa appender.
(Jerome
Boulon via cdouglas)
HADOOP-5264
.
Removes redundant configuration object from the TaskTracker.
(Sharad
Agarwal via ddas)
HADOOP-5232
.
Enable patch testing to occur on more than one host.
(Giri Kesavan
via nigel)
HADOOP-4546
.
Fix DF reporting for AIX.
(Bill Habermaas via cdouglas)
HADOOP-5023
.
Add Tomcat support to HdfsProxy.
(Zhiyong Zhang via cdouglas)
HADOOP-5317
.
Provide documentation for LazyOutput Feature.
(Jothi Padmanabhan via
johan)
HADOOP-5455
.
Document rpc metrics context to the extent dfs, mapred, and
jvm contexts are documented.
(Philip Zeyliger via cdouglas)
HADOOP-5358
.
Provide scripting functionality to the synthetic load
generator.
(Jakob Homan via hairong)
HADOOP-5442
.
Paginate jobhistory display and added some search
capabilities.
(Amar Kamat via acmurthy)
HADOOP-4842
.
Streaming now allows specifiying a command for the combiner.
(Amareshwari
Sriramadasu via ddas)
HADOOP-5196
.
avoiding unnecessary byte[] allocation in
SequenceFile.CompressedBytes and SequenceFile.UncompressedBytes.
(hong
tang via mahadev)
HADOOP-4655
.
New method FileSystem.newInstance() that always returns
a newly allocated FileSystem object.
(dhruba)
HADOOP-4788
.
Set Fair scheduler to assign both a map and a reduce on each
heartbeat by default.
(matei)
HADOOP-5491
.
In contrib/index, better control memory usage.
(Ning Li via
cutting)
HADOOP-5423
.
Include option of preserving file metadata in
SequenceFile::sort.
(Michael Tamm via cdouglas)
HADOOP-5331
.
Add support for KFS appends.
(Sriram Rao via cdouglas)
HADOOP-4365
.
Make Configuration::getProps protected in support of
meaningful subclassing.
(Steve Loughran via cdouglas)
HADOOP-2413
.
Remove the static variable FSNamesystem.fsNamesystemObject.
(Konstantin
Shvachko via szetszwo)
HADOOP-4584
.
Improve datanode block reports and associated file system
scan to avoid interefering with normal datanode operations.
(Suresh
Srinivas via rangadi)
HADOOP-5502
.
Documentation for backup and checkpoint nodes.
(Jakob Homan via shv)
HADOOP-5485
.
Mask actions in the fair scheduler's servlet UI based on
value of webinterface.private.actions.
(Vinod Kumar Vavilapalli via
yhemanth)
HADOOP-5581
.
HDFS should throw FileNotFoundException when while opening
a file that does not exist.
(Brian Bockelman via rangadi)
HADOOP-5509
.
PendingReplicationBlocks does not start monitor in the
constructor.
(shv)
HADOOP-5494
.
Modify sorted map output merger to lazily read values,
rather than buffering at least one record for each segment.
(Devaraj
Das
via cdouglas)
HADOOP-5396
.
Provide ability to refresh queue ACLs in the JobTracker
without having to restart the daemon.
(Sreekanth Ramakrishnan and
Vinod Kumar Vavilapalli via yhemanth)
HADOOP-4490
.
Provide ability to run tasks as job owners.
(Sreekanth Ramakrishnan
via yhemanth)
HADOOP-5697
.
Change org.apache.hadoop.examples.Grep to use new
mapreduce api.
(Amareshwari Sriramadasu via sharad)
HADOOP-5625
.
Add operation duration to clienttrace.
(Lei Xu via cdouglas)
HADOOP-5705
.
Improve TotalOrderPartitioner efficiency by updating the trie
construction.
(Dick King via cdouglas)
HADOOP-5589
.
Eliminate source limit of 64 for map-side joins imposed by
TupleWritable encoding.
(Jingkei Ly via cdouglas)
HADOOP-5734
.
Correct block placement policy description in HDFS
Design document.
(Konstantin Boudnik via shv)
HADOOP-5657
.
Validate data in TestReduceFetch to improve merge test
coverage.
(cdouglas)
HADOOP-5613
.
Change S3Exception to checked exception.
(Andrew Hitchcock via
tomwhite)
HADOOP-5717
.
Create public enum class for the Framework counters in
org.apache.hadoop.mapreduce.
(Amareshwari Sriramadasu via sharad)
HADOOP-5217
.
Split AllTestDriver for core, hdfs and mapred.
(sharad)
HADOOP-5364
.
Add certificate expiration warning to HsftpFileSystem and HDFS
proxy.
(Zhiyong Zhang via cdouglas)
HADOOP-5733
.
Add map/reduce slot capacity and blacklisted capacity to
JobTracker metrics.
(Sreekanth Ramakrishnan via cdouglas)
HADOOP-5596
.
Add EnumSetWritable.
(He Yongqiang via szetszwo)
HADOOP-5727
.
Simplify hashcode for ID types.
(Shevek via cdouglas)
HADOOP-5500
.
In DBOutputFormat, where field names are absent permit the
number of fields to be sufficient to construct the select query.
(Enis
Soztutar
via cdouglas)
HADOOP-5081
.
Split TestCLI into HDFS, Mapred and Core tests.
(sharad)
HADOOP-5015
.
Separate block management code from FSNamesystem.
(Suresh
Srinivas via szetszwo)
HADOOP-5080
.
Add new test cases to TestMRCLI and TestHDFSCLI
(V.Karthikeyan via
nigel)
HADOOP-5135
.
Splits the tests into different directories based on the
package. Four new test targets have been defined - run-test-core,
run-test-mapred, run-test-hdfs and run-test-hdfs-with-mr.
(Sharad
Agarwal via ddas)
HADOOP-5771
.
Implements unit tests for LinuxTaskController.
(Sreekanth
Ramakrishnan and Vinod Kumar Vavilapalli via yhemanth)
HADOOP-5419
.
Provide a facility to query the Queue ACLs for the
current user.
(Rahul Kumar Singh via yhemanth)
HADOOP-5780
.
Improve per block message prited by "-metaSave" in HDFS.
(Raghu
Angadi)
HADOOP-5823
.
Added a new class DeprecatedUTF8 to help with removing
UTF8 related javac warnings. These warnings are removed in
FSEditLog.java as a use case.
(Raghu Angadi)
HADOOP-5824
.
Deprecate DataTransferProtocol.OP_READ_METADATA and remove
the corresponding unused codes.
(Kan Zhang via szetszwo)
HADOOP-5721
.
Factor out EditLogFileInputStream and EditLogFileOutputStream
into independent classes.
(Luca Telloli & Flavio Junqueira via
shv)
HADOOP-5838
.
Fix a few javac warnings in HDFS.
(Raghu Angadi)
HADOOP-5854
.
Fix a few "Inconsistent Synchronization" warnings in HDFS.
(Raghu
Angadi)
HADOOP-5369
.
Small tweaks to reduce MapFile index size.
(Ben Maurer
via sharad)
HADOOP-5858
.
Eliminate UTF8 and fix warnings in test/hdfs-with-mr package.
(shv)
HADOOP-5866
.
Move DeprecatedUTF8 from o.a.h.io to o.a.h.hdfs since it may
not be used outside hdfs.
(Raghu Angadi)
HADOOP-5857
.
Move normal java methods from hdfs .jsp files to .java files.
(szetszwo)
HADOOP-5873
.
Remove deprecated methods randomDataNode() and
getDatanodeByIndex(..) in FSNamesystem.
(szetszwo)
HADOOP-5572
.
Improves the progress reporting for the sort phase for both
maps and reduces.
(Ravi Gummadi via ddas)
HADOOP-5839
.
Fix EC2 scripts to allow remote job submission.
(Joydeep Sen Sarma
via tomwhite)
HADOOP-5877
.
Fix javac warnings in TestHDFSServerPorts, TestCheckpoint,
TestNameEditsConfig, TestStartup and TestStorageRestore.
(Jakob Homan
via shv)
HADOOP-5438
.
Provide a single FileSystem method to create or
open-for-append to a file.
(He Yongqiang via dhruba)
HADOOP-5472
.
Change DistCp to support globbing of input paths.
(Dhruba
Borthakur and Rodrigo Schmidt via szetszwo)
HADOOP-5175
.
Don't unpack libjars on classpath.
(Todd Lipcon via tomwhite)
HADOOP-5620
.
Add an option to DistCp for preserving modification and access
times.
(Rodrigo Schmidt via szetszwo)
HADOOP-5664
.
Change map serialization so a lock is obtained only where
contention is possible, rather than for each write.
(cdouglas)
HADOOP-5896
.
Remove the dependency of GenericOptionsParser on
Option.withArgPattern.
(Giridharan Kesavan and Sharad Agarwal via
sharad)
HADOOP-5784
.
Makes the number of heartbeats that should arrive a second
at the JobTracker configurable.
(Amareshwari Sriramadasu via ddas)
HADOOP-5955
.
Changes TestFileOuputFormat so that is uses LOCAL_MR
instead of CLUSTER_MR.
(Jothi Padmanabhan via das)
HADOOP-5948
.
Changes TestJavaSerialization to use LocalJobRunner
instead of MiniMR/DFS cluster.
(Jothi Padmanabhan via das)
HADOOP-2838
.
Add mapred.child.env to pass environment variables to
tasktracker's child processes.
(Amar Kamat via sharad)
HADOOP-5961
.
DataNode process understand generic hadoop command line
options (like -Ddfs.property=value).
(Raghu Angadi)
HADOOP-5938
.
Change org.apache.hadoop.mapred.jobcontrol to use new
api.
(Amareshwari Sriramadasu via sharad)
HADOOP-2141
.
Improves the speculative execution heuristic. The heuristic
is currently based on the progress-rates of tasks and the expected time
to complete. Also, statistics about trackers are collected, and
speculative
tasks are not given to the ones deduced to be slow.
(Andy Konwinski
and ddas)
HADOOP-5952
.
Change "-1 tests included" wording in test-patch.sh.
(Gary Murry via
szetszwo)
HADOOP-6106
.
Provides an option in ShellCommandExecutor to timeout
commands that do not complete within a certain amount of time.
(Sreekanth
Ramakrishnan via yhemanth)
HADOOP-5925
.
EC2 scripts should exit on error.
(tomwhite)
HADOOP-6109
.
Change Text to grow its internal buffer exponentially, rather
than the max of the current length and the proposed length to improve
performance reading large values.
(thushara wijeratna via cdouglas)
HADOOP-2366
.
Support trimmed strings in Configuration.
(Michele Catasta
via szetszwo)
HADOOP-6099
.
The RPC module can be configured to not send period pings.
The default behaviour of sending periodic pings remain unchanged.
(dhruba)
HADOOP-6142
.
Update documentation and use of harchives for relative paths
added in MAPREDUCE-739.
(Mahadev Konar via cdouglas)
HADOOP-6148
.
Implement a fast, pure Java CRC32 calculator which outperforms
java.util.zip.CRC32.
(Todd Lipcon and Scott Carey via szetszwo)
HADOOP-6146
.
Upgrade to JetS3t version 0.7.1.
(tomwhite)
HADOOP-6161
.
Add get/setEnum methods to Configuration.
(cdouglas)
HADOOP-6160
.
Fix releaseaudit target to run on specific directories.
(gkesavan)
HADOOP-6169
.
Removing deprecated method calls in TFile.
(hong tang via
mahadev)
HADOOP-6176
.
Add a couple package private methods to AccessTokenHandler
for testing.
(Kan Zhang via szetszwo)
HADOOP-6182
.
Fix ReleaseAudit warnings
(Giridharan Kesavan and Lee Tucker
via gkesavan)
HADOOP-6173
.
Change src/native/packageNativeHadoop.sh to package all
native library files.
(Hong Tang via szetszwo)
HADOOP-6184
.
Provide an API to dump Configuration in a JSON format.
(V.V.Chaitanya
Krishna via yhemanth)
HADOOP-6224
.
Add a method to WritableUtils performing a bounded read of an
encoded String.
(Jothi Padmanabhan via cdouglas)
HADOOP-6133
.
Add a caching layer to Configuration::getClassByName to
alleviate a performance regression introduced in a compatibility layer.
(Todd
Lipcon via cdouglas)
HADOOP-6252
.
Provide a method to determine if a deprecated key is set in
config file.
(Jakob Homan via suresh)
HADOOP-5879
.
Read compression level and strategy from Configuration for
gzip compression.
(He Yongqiang via cdouglas)
HADOOP-6216
.
Support comments in host files.
(Ravi Phulari and Dmytro
Molkov via szetszwo)
HADOOP-6217
.
Update documentation for project split.
(Corinne Chandel via
omalley)
HADOOP-6268
.
Add ivy jar to .gitignore.
(Todd Lipcon via cdouglas)
HADOOP-6270
.
Support deleteOnExit in FileContext.
(Suresh Srinivas via
szetszwo)
HADOOP-6233
.
Rename configuration keys towards API standardization and
backward compatibility.
(Jithendra Pandey via suresh)
HADOOP-6260
.
Add additional unit tests for FileContext util methods.
(Gary Murry via suresh).
HADOOP-6309
.
Change build.xml to run tests with java asserts.
(Eli
Collins via szetszwo)
HADOOP-6326
.
Hundson runs should check for AspectJ warnings and report
failure if any is present
(cos)
HADOOP-6329
.
Add build-fi directory to the ignore lists.
(szetszwo)
HADOOP-5107
.
Use Maven ant tasks to publish the subproject jars.
(Giridharan
Kesavan via omalley)
HADOOP-6343
.
Log unexpected throwable object caught in RPC.
(Jitendra Nath
Pandey via szetszwo)
HADOOP-6367
.
Removes Access Token implementation from common.
(Kan Zhang via
ddas)
HADOOP-6395
.
Upgrade some libraries to be consistent across common, hdfs,
and mapreduce.
(omalley)
HADOOP-6398
.
Build is broken after HADOOP-6395
patch has been applied
(cos)
HADOOP-6413
.
Move TestReflectionUtils to Common.
(Todd Lipcon via tomwhite)
HADOOP-6283
.
Improve the exception messages thrown by
FileUtil$HardLink.getLinkCount(..).
(szetszwo)
HADOOP-6279
.
Add Runtime::maxMemory to JVM metrics.
(Todd Lipcon via
cdouglas)
HADOOP-6305
.
Unify build property names to facilitate cross-projects
modifications
(cos)
HADOOP-6312
.
Remove unnecessary debug logging in Configuration constructor.
(Aaron
Kimball via cdouglas)
HADOOP-6366
.
Reduce ivy console output to ovservable level
(cos)
HADOOP-6400
.
Log errors getting Unix UGI.
(Todd Lipcon via tomwhite)
HADOOP-6346
.
Add support for specifying unpack pattern regex to
RunJar.unJar.
(Todd Lipcon via tomwhite)
HADOOP-6422
.
Make RPC backend plugable, protocol-by-protocol, to
ease evolution towards Avro.
(cutting)
HADOOP-5958
.
Use JDK 1.6 File APIs in DF.java wherever possible.
(Aaron Kimball
via tomwhite)
HADOOP-6222
.
Core doesn't have TestCommonCLI facility.
(cos)
HADOOP-6394
.
Add a helper class to simplify FileContext related tests and
improve code reusability.
(Jitendra Nath Pandey via suresh)
HADOOP-4656
.
Add a user to groups mapping service.
(boryas, acmurthy)
HADOOP-6435
.
Make RPC.waitForProxy with timeout public.
(Steve Loughran
via tomwhite)
HADOOP-6472
.
add tokenCache option to GenericOptionsParser for passing
file with secret keys to a map reduce job.
(boryas)
HADOOP-3205
.
Read multiple chunks directly from FSInputChecker subclass
into user buffers.
(Todd Lipcon via tomwhite)
HADOOP-6479
.
TestUTF8 assertions could fail with better text.
(Steve Loughran via
tomwhite)
HADOOP-6155
.
Deprecate RecordIO anticipating Avro.
(Tom White via cdouglas)
HADOOP-6492
.
Make some Avro serialization APIs public.
(Aaron Kimball via
cutting)
HADOOP-6497
.
Add an adapter for Avro's SeekableInput interface, so
that Avro can read FileSystem data.
(Aaron Kimball via cutting)
HADOOP-6495
.
Identifier should be serialized after the password is
created In Token constructor
(jnp via boryas)
HADOOP-6518
.
Makes the UGI honor the env var KRB5CCNAME.
(Owen O'Malley via ddas)
HADOOP-6531
.
Enhance FileUtil with an API to delete all contents of a
directory.
(Amareshwari Sriramadasu via yhemanth)
HADOOP-6547
.
Move DelegationToken into Common, so that it can be used by
MapReduce also.
(devaraj via omalley)
HADOOP-6552
.
Puts renewTGT=true and useTicketCache=true for the keytab
kerberos options.
(ddas)
HADOOP-6534
.
Trim whitespace from directory lists initializing
LocalDirAllocator.
(Todd Lipcon via cdouglas)
HADOOP-6559
.
Makes the RPC client automatically re-login when the SASL
connection setup fails. This is applicable only to keytab based logins.
(Devaraj
Das)
HADOOP-6551
.
Delegation token renewing and cancelling should provide
meaningful exceptions when there are failures instead of returning
false.
(omalley)
HADOOP-6583
.
Captures authentication and authorization metrics.
(ddas)
HADOOP-6543
.
Allows secure clients to talk to unsecure clusters.
(Kan Zhang via
ddas)
HADOOP-6579
.
Provide a mechanism for encoding/decoding Tokens from
a url-safe string and change the commons-code library to 1.4.
(omalley)
HADOOP-6596
.
Add a version field to the AbstractDelegationTokenIdentifier's
serialized value.
(omalley)
HADOOP-6573
.
Support for persistent delegation tokens.
(Jitendra Pandey via shv)
HADOOP-6594
.
Provide a fetchdt tool via bin/hdfs.
(jhoman via acmurthy)
HADOOP-6589
.
Provide better error messages when RPC authentication fails.
(Kan
Zhang via omalley)
HADOOP-6599
Split existing RpcMetrics into RpcMetrics & RpcDetailedMetrics.
(Suresh
Srinivas via Sanjay Radia)
HADOOP-6537
Declare more detailed exceptions in FileContext and
AbstractFileSystem
(Suresh Srinivas via Sanjay Radia)
HADOOP-6486
.
fix common classes to work with Avro 1.3 reflection.
(cutting via
tomwhite)
HADOOP-6591
.
HarFileSystem can handle paths with the whitespace characters.
(Rodrigo
Schmidt via dhruba)
HADOOP-6407
.
Have a way to automatically update Eclipse .classpath file
when new libs are added to the classpath through Ivy.
(tomwhite)
HADOOP-3659
.
Patch to allow hadoop native to compile on Mac OS X.
(Colin Evans
and Allen Wittenauer via tomwhite)
HADOOP-6471
.
StringBuffer -> StringBuilder - conversion of references
as necessary.
(Kay Kay via tomwhite)
HADOOP-6646
.
Move HarfileSystem out of Hadoop Common.
(mahadev)
HADOOP-6566
.
Add methods supporting, enforcing narrower permissions on
local daemon directories.
(Arun Murthy and Luke Lu via cdouglas)
HADOOP-6705
.
Fix to work with 1.5 version of jiracli
(Giridharan Kesavan)
HADOOP-6658
.
Exclude Private elements from generated Javadoc.
(tomwhite)
HADOOP-6635
.
Install/deploy source jars to Maven repo.
(Patrick Angeles via
jghoman)
HADOOP-6717
.
Log levels in o.a.h.security.Groups too high
(Todd Lipcon via
jghoman)
HADOOP-6667
.
RPC.waitForProxy should retry through NoRouteToHostException.
(Todd
Lipcon via tomwhite)
HADOOP-6677
.
InterfaceAudience.LimitedPrivate should take a string not an
enum.
(tomwhite)
HADOOP-678
.
Remove FileContext#isFile, isDirectory, and exists.
(Eli Collins via
hairong)
HADOOP-6515
.
Make maximum number of http threads configurable.
(Scott Chen via
zshao)
HADOOP-6563
.
Add more symlink tests to cover intermediate symlinks
in paths.
(Eli Collins via suresh)
HADOOP-6585
.
Add FileStatus#isDirectory and isFile.
(Eli Collins via
tomwhite)
HADOOP-6738
.
Move cluster_setup.xml from MapReduce to Common.
(Tom White via
tomwhite)
HADOOP-6794
.
Move configuration and script files post split.
(tomwhite)
HADOOP-6403
.
Deprecate EC2 bash scripts.
(tomwhite)
HADOOP-6769
.
Add an API in FileSystem to get FileSystem instances based
on users
(ddas via boryas)
HADOOP-6813
.
Add a new newInstance method in FileSystem that takes
a "user" as argument
(ddas via boryas)
HADOOP-6668
.
Apply audience and stability annotations to classes in
common.
(tomwhite)
HADOOP-6821
.
Document changes to memory monitoring.
(Hemanth Yamijala
via tomwhite)
OPTIMIZATIONS
(12)
HADOOP-5595
.
NameNode does not need to run a replicator to choose a
random DataNode.
(hairong)
HADOOP-5603
.
Improve NameNode's block placement performance.
(hairong)
HADOOP-5638
.
More improvement on block placement performance.
(hairong)
HADOOP-6180
.
NameNode slowed down when many files with same filename
were moved to Trash.
(Boris Shkolnik via hairong)
HADOOP-6166
.
Further improve the performance of the pure-Java CRC32
implementation. (Tsz Wo (Nicholas), SZE via cdouglas)
HADOOP-6271
.
Add recursive and non recursive create and mkdir to
FileContext.
(Sanjay Radia via suresh)
HADOOP-6261
.
Add URI based tests for FileContext.
(Ravi Pulari via suresh).
HADOOP-6307
.
Add a new SequenceFile.Reader constructor in order to support
reading on un-closed file.
(szetszwo)
HADOOP-6467
.
Improve the performance on HarFileSystem.listStatus(..).
(mahadev
via szetszwo)
HADOOP-6569
.
FsShell#cat should avoid calling unecessary getFileStatus
before opening a file to read.
(hairong)
HADOOP-6689
.
Add directory renaming test to existing FileContext tests.
(Eli
Collins via suresh)
HADOOP-6713
.
The RPC server Listener thread is a scalability bottleneck.
(Dmytro
Molkov via hairong)
BUG FIXES
(231)
不过，个人觉得hadoop的稳定性还是比hbase好，虽然hbase更新速度无比的快。
算了，还是言归正传，升级：
1、core-site.xml
不变
2、hdfs.site.xml
可以不变也可以把dfs.name.dir和dfs.data.dir更新成这样：
<property>
<name>dfs.namenode.name.dir</name>
<value>/data0/cloud/namenode/</value>
</description>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>/data0/cloud/datanode/</value>
</property>
不过，有个警告就是：
WARN org.apache.hadoop.hdfs.server.common.Util: Path /data0/cloud/namenode/ should be specified as a URI in configuration files. Please update hdfs configuration.
3、mapred-site.xml
<!--
<property>
<name>mapred.job.tracker</name>
<value>name.uc.uuwatch.com:9001</value>
</property>
-->
<property>
<name>mapreduce.jobtracker.address</name>
<value>tracker.uc.uuwatch.com:9001</value>
</property>
同步所有的配置文件到到所有的节点，然后启动hadoop
...bin/start-all.sh
你会发现脚本也更新了，比之前的分类更好，控制更加细致，很好！
不过，可惜的是namenode和secondnamenode不能启动，分析下日志会发现需要升级文件（原来文件格式也有一定的更改，优化了吗，呵呵）：
ERROR org.apache.hadoop.hdfs.server.namenode.FSNamesystem: FSNamesystem initialization failed.
java.io.IOException:
File system image contains an old layout version -18.
An upgrade to version -24 is required.
Please restart NameNode with -upgrade option.
4、OK 行了，执行名称节点的文件系统更新命令
吧
.../bin/hadoop namenode -upgrade
好，到此成功启动节点。升级完毕！
接下来，我启动hbase0.20.6试试了。
特别提醒：备份你的hadoop数据，不要回不到原来的版本了。请查阅 bin/hadoop namenode 相关的命令以及数据节点的备份。
简单点：可以拷贝你的原始数据到新的文件夹，更改配置，使用新文件夹的数据升级尝试，失败了也不怕。数据完好
hbase0.20.6是基于hadoop0.20.2的 发现升级后 RPC对不住，文件系统对不住。
更改hbade的依赖，改RPC 去叉 重新编译 部署
另外：经过修改hbase可以启动 但是不能读写表，还没有修改完毕的原因。 |
|
QQ群⑧：
窥视卡
雷达卡
发表于 2016-12-8 09:16:01
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡