HBase官方指南（2015最新版本）-配置详解篇

HBase最新官方指南的原创翻译，原文地址http://hbase.apache.org/book.html。 本部分目录如下： Apache HBase 配置 9 3.配置文件 9 4.先决条件（当然是部署HBase的咯） 11 JDK环境 表2. Java 11 操作系统基础服务 12 ssh 12 DNS 12 Loopback IP 12 NTP 13 文件和进程的数量限制（与文件系统及程序的限制关系） 13 Linux shell 15 Windows 15 4.1 Hadoop 15 4.1.1 Apache HBase 0.94和Hadoop2 18 4.1.2 Apache HBase0.92和0.94 19 4.1.3 Apache HBase 0.96 19 4.1.4 Hadoop 0.20.x-1.x 20 4.1.5 运行在安全的Hadoop上的HBase 20 4.1.6关于 dfs.datanode.max.transfer.threads 21 4.2 Zookeeper 21 5.HBase的运行模式：独立运行与分布式运行 22 5.1HBase独立运行模式 22 5.2HBase分布式模式 22 5.2.1伪分布式（ Pseudo-distributed） 23 5.3完全分布式（Fully-distributed） 23 分布式的RegionServer 23 Zookeeper与HBase 24 分布式HBase快速入门 25 HDFS客户端配置步骤 25 6.运行验证你的安装 25 7.默认配置 27 7.1hbase-site.xml和hbase-default.xml 27 7.2HBase默认配置参数说明 27 hbase.tmp.dir 27 hbase.rootdir 28 hbase.fs.tmp.dir 28 hbase.bulkload.staging.dir 28 hbase.cluster.distributed 29 Hbase.zookeeper.quorum 29 hbase.local.dir 29 hbase.master.port 30 hbase.master.info.port 30 hbase.master.info.bindAddress 30 hbase.master.logcleaner.plugins 30 hbase.master.logcleaner.ttl 31 hbase.master.hfilecleaner.plugins 31 hbase.master.catalog.timeout 31 hbase.master.infoserver.redirect 32 hbase.regionserver.port 32 hbase.regionserver.info.port 32 hbase.regionserver.info.bindAddress 32 hbase.regionserver.info.port.auto 33 hbase.regionserver.handler.count 33 hbase.ipc.server.callqueue.handler.factor 33 hbase.ipc.server.callqueue.read.ratio 34 hbase.ipc.server.callqueue.scan.ratio 35 hbase.regionserver.msginterval 36 hbase.regionserver.logroll.period 36 hbase.regionserver.logroll.errors.tolerated 36 hbase.regionserver.hlog.reader.impl 37 hbase.regionserver.hlog.writer.impl 37 hbase.regionserver.global.memstore.size 37 hbase.regionserver.global.memstore.size.lower.limit 38 hbase.regionserver.optionalcacheflushinterval 38 hbase.regionserver.catalog.timeout 39 hbase.regionserver.dns.interface 39 hbase.regionserver.dns.nameserver 39 hbase.regionserver.region.split.policy 40 hbase.regionserver.regionSplitLimit 40 zookeeper.session.timeout 41 zookeeper.znode.parent 42 zookeeper.znode.rootserver 42 zookeeper.znode.acl.parent 43 hbase.zookeeper.dns.interface 43 hbase.zookeeper.dns.nameserver 43 hbase.zookeeper.peerport 44 hbase.zookeeper.leaderport 44 hbase.zookeeper.useMulti 44 hbase.zookeeper.property.initLimit 45 hbase.zookeeper.property.syncLimit 45 hbase.zookeeper.property.dataDir 46 hbase.zookeeper.property.clientPort 46 hbase.zookeeper.property.maxClientCnxns 46 hbase.client.write.buffer 47 hbase.client.pause 47 hbase.client.retries.number 48 hbase.client.max.total.tasks 48 hbase.client.max.perserver.tasks 49 hbase.client.max.perregion.tasks 49 hbase.client.scanner.caching 49 hbase.client.keyvalue.maxsize 50 hbase.client.scanner.timeout.period 51 hbase.client.localityCheck.threadPoolSize 51 hbase.bulkload.retries.number 51 hbase.balancer.period 52 hbase.normalizer.enabled 52 hbase.normalizer.period 52 hbase.regions.slop 53 hbase.server.thread.wakefrequency 53 hbase.server.versionfile.writeattempts 53 hbase.hregion.memstore.flush.size 54 hbase.hregion.percolumnfamilyflush.size.lower.bound.min 54 hbase.hregion.preclose.flush.size 55 hbase.hregion.memstore.block.multiplier 56 hbase.hregion.memstore.mslab.enabled 56 hbase.hregion.max.filesize 57 hbase.hregion.majorcompaction 57 hbase.hregion.majorcompaction.jitter 58 hbase.hstore.compactionThreshold 59 hbase.hstore.flusher.count 59 hbase.hstore.blockingStoreFiles 60 hbase.hstore.blockingWaitTime 60 hbase.hstore.compaction.min 61 hbase.hstore.compaction.max 61 hbase.hstore.compaction.min.size 61 hbase.hstore.compaction.max.size 62 hbase.hstore.compaction.ratio 62 hbase.hstore.compaction.ratio.offpeak 63 hbase.hstore.time.to.purge.deletes 63 hbase.offpeak.start.hour 63 hbase.offpeak.end.hour 63 hbase.regionserver.thread.compaction.throttle 64 hbase.hstore.compaction.kv.max 64 hbase.storescanner.parallel.seek.enable 64 hbase.storescanner.parallel.seek.threads 65 hfile.block.cache.size 65 hfile.block.index.cacheonwrite 65 hfile.index.block.max.size 65 hbase.bucketcache.ioengine 66 hbase.bucketcache.combinedcache.enabled 66 hbase.bucketcache.size 66 hbase.bucketcache.sizes 66 hfile.format.version 67 hfile.block.bloom.cacheonwrite 67 io.storefile.bloom.block.size 67 hbase.rs.cacheblocksonwrite 68 hbase.rpc.timeout 68 hbase.cells.scanned.per.heartbeat.check 68 hbase.rpc.shortoperation.timeout 68 hbase.ipc.client.tcpnodelay 69 hbase.regionserver.hostname 69 hbase.master.keytab.file 69 hbase.master.kerberos.principal 70 hbase.regionserver.keytab.file 70 hbase.regionserver.kerberos.principal 70 hadoop.policy.file 70 hbase.superuser 71 hbase.auth.key.update.interval 71 hbase.auth.token.max.lifetime 71 hbase.ipc.client.fallback-to-simple-auth-allowed 71 hbase.ipc.server.fallback-to-simple-auth-allowed 72 hbase.display.keys 72 hbase.coprocessor.enabled 72 hbase.coprocessor.user.enabled 73 hbase.coprocessor.region.classes 73 hbase.rest.port 73 hbase.rest.readonly 74 hbase.rest.threads.max 74 hbase.rest.threads.min 74 hbase.rest.support.proxyuser 74 hbase.defaults.for.version.skip 75 hbase.coprocessor.master.classes 75 hbase.coprocessor.abortonerror 75 hbase.online.schema.update.enable 76 hbase.table.lock.enable 76 hbase.table.max.rowsize 76 hbase.thrift.minWorkerThreads 76 hbase.thrift.maxWorkerThreads 77 hbase.thrift.maxQueuedRequests 77 hbase.thrift.htablepool.size.max 77 hbase.regionserver.thrift.framed 78 hbase.regionserver.thrift.framed.max_frame_size_in_mb 78 hbase.regionserver.thrift.compact 78 hbase.rootdir.perms 78 hbase.data.umask.enable 79 hbase.data.umask 79 hbase.metrics.showTableName 79 hbase.metrics.exposeOperationTimes 79 hbase.snapshot.enabled 80 hbase.snapshot.restore.take.failsafe.snapshot 80 hbase.snapshot.restore.failsafe.name 80 hbase.server.compactchecker.interval.multiplier 80 hbase.lease.recovery.timeout 81 hbase.lease.recovery.dfs.timeout 81 hbase.column.max.version 81 dfs.client.read.shortcircuit 82 dfs.domain.socket.path 82 hbase.dfs.client.read.shortcircuit.buffer.size 82 hbase.regionserver.checksum.verify 82 hbase.hstore.bytes.per.checksum 83 hbase.hstore.checksum.algorithm 83 hbase.client.scanner.max.result.size 83 hbase.server.scanner.max.result.size 84 hbase.status.published 84 hbase.status.publisher.class 84 hbase.status.listener.class 85 hbase.status.multicast.address.ip 85 hbase.status.multicast.address.port 85 hbase.dynamic.jars.dir 85 hbase.security.authentication 86 hbase.rest.filter.classes 86 hbase.master.loadbalancer.class 86 hbase.master.normalizer.class 86 hbase.security.exec.permission.checks 87 hbase.procedure.regionserver.classes 87 hbase.procedure.master.classes 87 hbase.coordinated.state.manager.class 88 hbase.regionserver.storefile.refresh.period 88 hbase.region.replica.replication.enabled 88 hbase.http.filter.initializers 89 hbase.security.visibility.mutations.checkauths 89 hbase.http.max.threads 89 hbase.replication.rpc.codec 90 hbase.replication.source.maxthreads 90 hbase.http.staticuser.user 90 hbase.regionserver.handler.abort.on.error.percent 91 hbase.mob.file.cache.size 91 hbase.mob.cache.evict.period 91 hbase.mob.cache.evict.remain.ratio 91 hbase.mob.sweep.tool.compaction.ratio 92 hbase.mob.sweep.tool.compaction.mergeable.size 92 hbase.mob.sweep.tool.compaction.memstore.flush.size 92 hbase.master.mob.ttl.cleaner.period 93 hbase.mob.compaction.mergeable.threshold 93 hbase.mob.delfile.max.count 93 hbase.mob.compaction.batch.size 94 hbase.mob.compaction.chore.period 94 hbase.mob.compactor.class 95 hbase.mob.compaction.threads.max 95 hbase.snapshot.master.timeout.millis 95 hbase.snapshot.region.timeout 96 7.3 hbase-env.sh 96 7.4 log4j.properties 96 7.5 客户端连接到HBase的配置和依赖 97 7.5.1 java client 端配置 98 8.配置实例 98 8.1HBase基本的分布式配置 98 8.1.1hbase-site.xml 98 8.1.2. regionservers 99 8.1.3. hbase-env.sh 100 9.重要的配置 100 9.1必须的配置 100 9.1.1 大集群配置 101 9.1.2 如果有backup Master，请将primary Master配置为fail fast 101 9.2推荐配置 101 9.2.1zookeeper 配置 101 zookeeper.session.timeout 101 Number of ZooKeeper Instances 102 9.2.2 HDFS配置 102 dfs.datanode.failed.volumes.tolerated 102 9.2.3. hbase.regionserver.handler.count 102 9.2.4. 配置大内存的机器 103 9.2.5压缩 103 9.2.6. 配置WAL 文件的数量和大小 103 9.2.7. Managed Splitting（管理region切分） 104 9.2.8. 管理合并（Managed Compactions） 105 9.2.9. 预测执行（Speculative Execution） 106 9.3. 其他配置（Other Configurations） 106 9.3.1. 负载均衡（Balancer） 106 9.3.2. 禁止块缓存（Disabling Blockcache） 106 9.3.3. Nagle算法或小包问题（Nagle’s or the small package problem） 107 9.3.4.更好的平均恢复时间（ Better Mean Time to Recover (MTTR)） 107 9.3.5. JMX 109 10. 动态配置（Dynamic Configuration） 111 If your organization is looking for a storage solution to accommodate a virtually endless amount of data, this book will show you how Apache HBase can fulfill your needs. As the open source implementation of Google's BigTable architecture, HBase scales to billions of rows and millions of columns, while ensuring that write and read performance remain constant. HBase: The Definitive Guide provides the details you require, whether you simply want to evaluate this high-performance, non-relational database, or put it into practice right away. HBase's adoption rate is beginning to climb, and several IT executives are asking pointed questions about this high-capacity database. This is the only book available to give you meaningful answers. * Learn how to distribute large datasets across an inexpensive cluster of commodity servers * Develop HBase clients in many programming languages, including Java, Python, and Ruby * Get details on HBase's primary storage system, HDFS-Hadoop's distributed and replicated filesystem * Learn how HBase's native interface to Hadoop's MapReduce framework enables easy development and execution of batch jobs that can scan entire tables * Discover the integration between HBase and other facets of the Apache Hadoop project Pseudo-distributed） 23 5.3完全分布式（Fully-distributed） 23 分布式的RegionServer 23 Zookeeper与HBase 24 分布式HBase快速入门 25 HDFS客户端配置步骤 25 6.运行验证你的安装 25 7.默认配置 27 7.1hbase-site.xml和hbase-default.xml 27 7.2HBase默认配置参数说明 27 hbase.tmp.dir 27 hbase.rootdir 28 hbase.fs.tmp.dir 28 hbase.bulkload.staging.dir 28 hbase.cluster.distributed 29 Hbase.zookeeper.quorum 29 hbase.local.dir 29 hbase.master.port 30 hbase.master.info.port 30 hbase.master.info.bindAddress 30 hbase.master.logcleaner.plugins 30 hbase.master.logcleaner.ttl 31 hbase.master.hfilecleaner.plugins 31 hbase.master.catalog.timeout 31 hbase.master.infoserver.redirect 32 hbase.regionserver.port 32 hbase.regionserver.info.port 32 hbase.regionserver.info.bindAddress 32 hbase.regionserver.info.port.auto 33 hbase.regionserver.handler.count 33 hbase.ipc.server.callqueue.handler.factor 33 hbase.ipc.server.callqueue.read.ratio 34 hbase.ipc.server.callqueue.scan.ratio 35 hbase.regionserver.msginterval 36 hbase.regionserver.logroll.period 36 hbase.regionserver.logroll.errors.tolerated 36 hbase.regionserver.hlog.reader.impl 37 hbase.regionserver.hlog.writer.impl 37 hbase.regionserver.global.memstore.size 37 hbase.regionserver.global.memstore.size.lower.limit 38 hbase.regionserver.optionalcacheflushinterval 38 hbase.regionserver.catalog.timeout 39 hbase.regionserver.dns.interface 39 hbase.regionserver.dns.nameserver 39 hbase.regionserver.region.split.policy 40 hbase.regionserver.regionSplitLimit 40 zookeeper.session.timeout 41 zookeeper.znode.parent 42 zookeeper.znode.rootserver 42 zookeeper.znode.acl.parent 43 hbase.zookeeper.dns.interface 43 hbase.zookeeper.dns.nameserver 43 hbase.zookeeper.peerport 44 hbase.zookeeper.leaderport 44 hbase.zookeeper.useMulti 44 hbase.zookeeper.property.initLimit 45 hbase.zookeeper.property.syncLimit 45 hbase.zookeeper.property.dataDir 46 hbase.zookeeper.property.clientPort 46 hbase.zookeeper.property.maxClientCnxns 46 hbase.client.write.buffer 47 hbase.client.pause 47 hbase.client.retries.number 48 hbase.client.max.total.tasks 48 hbase.client.max.perserver.tasks 49 hbase.client.max.perregion.tasks 49 hbase.client.scanner.caching 49 hbase.client.keyvalue.maxsize 50 hbase.client.scanner.timeout.period 51 hbase.client.localityCheck.threadPoolSize 51 hbase.bulkload.retries.number 51 hbase.balancer.period 52 hbase.normalizer.enabled 52 hbase.normalizer.period 52 hbase.regions.slop 53 hbase.server.thread.wakefrequency 53 hbase.server.versionfile.writeattempts 53 hbase.hregion.memstore.flush.size 54 hbase.hregion.percolumnfamilyflush.size.lower.bound.min 54 hbase.hregion.preclose.flush.size 55 hbase.hregion.memstore.block.multiplier 56 hbase.hregion.memstore.mslab.enabled 56 hbase.hregion.max.filesize 57 hbase.hregion.majorcompaction 57 hbase.hregion.majorcompaction.jitter 58 hbase.hstore.compactionThreshold 59 hbase.hstore.flusher.count 59 hbase.hstore.blockingStoreFiles 60 hbase.hstore.blockingWaitTime 60 hbase.hstore.compaction.min 61 hbase.hstore.compaction.max 61 hbase.hstore.compaction.min.size 61 hbase.hstore.compaction.max.size 62 hbase.hstore.compaction.ratio 62 hbase.hstore.compaction.ratio.offpeak 63 hbase.hstore.time.to.purge.deletes 63 hbase.offpeak.start.hour 63 hbase.offpeak.end.hour 63 hbase.regionserver.thread.compaction.throttle 64 hbase.hstore.compaction.kv.max 64 hbase.storescanner.parallel.seek.enable 64 hbase.storescanner.parallel.seek.threads 65 hfile.block.cache.size 65 hfile.block.index.cacheonwrite 65 hfile.index.block.max.size 65 hbase.bucketcache.ioengine 66 hbase.bucketcache.combinedcache.enabled 66 hbase.bucketcache.size 66 hbase.bucketcache.sizes 66 hfile.format.version 67 hfile.block.bloom.cacheonwrite 67 io.storefile.bloom.block.size 67 hbase.rs.cacheblocksonwrite 68 hbase.rpc.timeout 68 hbase.cells.scanned.per.heartbeat.check 68 hbase.rpc.shortoperation.timeout 68 hbase.ipc.client.tcpnodelay 69 hbase.regionserver.hostname 69 hbase.master.keytab.file 69 hbase.master.kerberos.principal 70 hbase.regionserver.keytab.file 70 hbase.regionserver.kerberos.principal 70 hadoop.policy.file 70 hbase.superuser 71 hbase.auth.key.update.interval 71 hbase.auth.token.max.lifetime 71 hbase.ipc.client.fallback-to-simple-auth-allowed 71 hbase.ipc.server.fallback-to-simple-auth-allowed 72 hbase.display.keys 72 hbase.coprocessor.enabled 72 hbase.coprocessor.user.enabled 73 hbase.coprocessor.region.classes 73 hbase.rest.port 73 hbase.rest.readonly 74 hbase.rest.threads.max 74 hbase.rest.threads.min 74 hbase.rest.support.proxyuser 74 hbase.defaults.for.version.skip 75 hbase.coprocessor.master.classes 75 hbase.coprocessor.abortonerror 75 hbase.online.schema.update.enable 76 hbase.table.lock.enable 76 hbase.table.max.rowsize 76 hbase.thrift.minWorkerThreads 76 hbase.thrift.maxWorkerThreads 77 hbase.thrift.maxQueuedRequests 77 hbase.thrift.htablepool.size.max 77 hbase.regionserver.thrift.framed 78 hbase.regionserver.thrift.framed.max_frame_size_in_mb 78 hbase.regionserver.thrift.compact 78 hbase.rootdir.perms 78 hbase.data.umask.enable 79 hbase.data.umask 79 hbase.metrics.showTableName 79 hbase.metrics.exposeOperationTimes 79 hbase.snapshot.enabled 80 hbase.snapshot.restore.take.failsafe.snapshot 80 hbase.snapshot.restore.failsafe.name 80 hbase.server.compactchecker.interval.multiplier 80 hbase.lease.recovery.timeout 81 hbase.lease.recovery.dfs.timeout 81 hbase.column.max.version 81 dfs.client.read.shortcircuit 82 dfs.domain.socket.path 82 hbase.dfs.client.read.shortcircuit.buffer.size 82 hbase.regionserver.checksum.verify 82 hbase.hstore.bytes.per.checksum 83 hbase.hstore.checksum.algorithm 83 hbase.client.scanner.max.result.size 83 hbase.server.scanner.max.result.size 84 hbase.status.published 84 hbase.status.publisher.class 84 hbase.status.listener.class 85 hbase.status.multicast.address.ip 85 hbase.status.multicast.address.port 85 hbase.dynamic.jars.dir 85 hbase.security.authentication 86 hbase.rest.filter.classes 86 hbase.master.loadbalancer.class 86 hbase.master.normalizer.class 86 hbase.security.exec.permission.checks 87 hbase.procedure.regionserver.classes 87 hbase.procedure.master.classes 87 hbase.coordinated.state.manager.class 88 hbase.regionserver.storefile.refresh.period 88 hbase.region.replica.replication.enabled 88 hbase.http.filter.initializers 89 hbase.security.visibility.mutations.checkauths 89 hbase.http.max.threads 89 hbase.replication.rpc.codec 90 hbase.replication.source.maxthreads 90 hbase.http.staticuser.user 90 hbase.regionserver.handler.abort.on.error.percent 91 hbase.mob.file.cache.size 91 hbase.mob.cache.evict.period 91 hbase.mob.cache.evict.remain.ratio 91 hbase.mob.sweep.tool.compaction.ratio 92 hbase.mob.sweep.tool.compaction.mergeable.size 92 hbase.mob.sweep.tool.compaction.memstore.flush.size 92 hbase.master.mob.ttl.cleaner.period 93 hbase.mob.compaction.mergeable.threshold 93 hbase.mob.delfile.max.count 93 hbase.mob.compaction.batch.size 94 hbase.mob.compaction.chore.period 94 hbase.mob.compactor.class 95 hbase.mob.compaction.threads.max 95 hbase.snapshot.master.timeout.millis 95 hbase.snapshot.region.timeout 96 7.3 hbase-env.sh 96 7.4 log4j.properties 96 7.5 客户端连接到HBase的配置和依赖 97 7.5.1 java client 端配置 98 8.配置实例 98 8.1HBase基本的分布式配置 98 8.1.1hbase-site.xml 98 8.1.2. regionservers 99 8.1.3. hbase-env.sh 100 9.重要的配置 100 9.1必须的配置 100 9.1.1 大集群配置 101 9.1.2 如果有backup Master，请将primary Master配置为fail fast 101 9.2推荐配置 101 9.2.1zookeeper 配置 101 zookeeper.session.timeout 101 Number of ZooKeeper Instances 102 9.2.2 HDFS配置 102 dfs.datanode.failed.volumes.tolerated 102 9.2.3. hbase.regionserver.handler.count 102 9.2.4. 配置大内存的机器 103 9.2.5压缩 103 9.2.6. 配置WAL 文件的数量和大小 103 9.2.7. Managed Splitting（管理region切分） 104 9.2.8. 管理合并（Managed Compactions） 105 9.2.9. 预测执行（Speculative Execution） 106 9.3. 其他配置（Other Configurations） 106 9.3.1. 负载均衡（Balancer） 106 9.3.2. 禁止块缓存（Disabling Blockcache） 106 9.3.3. Nagle算法或小包问题（Nagle’s or the small package problem） 107 9.3.4.更好的平均恢复时间（ Better Mean Time to Recover (MTTR)） 107 9.3.5. JMX 109 10. 动态配置（Dynamic Configuration） 111 If your organization is looking for a storage solution to accommodate a virtually endless amount of data, this book will show you how Apache HBase can fulfill your needs. As the open source implementation of Google's BigTable architecture, HBase scales to billions of rows and millions of columns, while ensuring that write and read performance remain constant. HBase: The Definitive Guide provides the details you require, whether you simply want to evaluate this high-performance, non-relational database, or put it into practice right away. HBase's adoption rate is beginning to climb, and several IT executives are asking pointed questions about this high-capacity database. This is the only book available to give you meaningful answers. * Learn how to distribute large datasets across an inexpensive cluster of commodity servers * Develop HBase clients in many programming languages, including Java, Python, and Ruby * Get details on HBase's primary storage system, HDFS-Hadoop's distributed and replicated filesystem * Learn how HBase's native interface to Hadoop's MapReduce framework enables easy development and execution of batch jobs that can scan entire tables * Discover the integration between HBase and other facets of the Apache Hadoop project