Class CompactionManager

  • All Implemented Interfaces:
    CompactionManagerMBean

    public class CompactionManager
    extends java.lang.Object
    implements CompactionManagerMBean

    A singleton which manages a private executor of ongoing compactions.

    Scheduling for compaction is accomplished by swapping sstables to be compacted into a set via Tracker. New scheduling attempts will ignore currently compacting sstables.
    • Field Detail

      • MBEAN_OBJECT_NAME

        public static final java.lang.String MBEAN_OBJECT_NAME
        See Also:
        Constant Field Values
      • currentlyBackgroundUpgrading

        public final java.util.concurrent.atomic.AtomicInteger currentlyBackgroundUpgrading
      • isCompactionManager

        public static final io.netty.util.concurrent.FastThreadLocal<java.lang.Boolean> isCompactionManager
    • Constructor Detail

      • CompactionManager

        public CompactionManager()
    • Method Detail

      • getRateLimiter

        public com.google.common.util.concurrent.RateLimiter getRateLimiter()
        Gets compaction rate limiter. Rate unit is bytes per sec.
        Returns:
        RateLimiter with rate limit set
      • setRate

        public void setRate​(double throughPutMbPerSec)
        Sets the rate for the rate limiter. When compaction_throughput_mb_per_sec is 0 or node is bootstrapping, this sets the rate to Double.MAX_VALUE bytes per second.
        Parameters:
        throughPutMbPerSec - throughput to set in mb per second
      • submitBackground

        public java.util.List<java.util.concurrent.Future<?>> submitBackground​(ColumnFamilyStore cfs)
        Call this whenever a compaction might be needed on the given columnfamily. It's okay to over-call (within reason) if a call is unnecessary, it will turn into a no-op in the bucketing/candidate-scan phase.
      • isCompacting

        public boolean isCompacting​(java.lang.Iterable<ColumnFamilyStore> cfses,
                                    java.util.function.Predicate<SSTableReader> sstablePredicate)
      • forceShutdown

        public void forceShutdown()
        Shutdowns both compaction and validation executors, cancels running compaction / validation, and waits for tasks to complete if tasks were not cancelable.
      • finishCompactionsAndShutdown

        public void finishCompactionsAndShutdown​(long timeout,
                                                 java.util.concurrent.TimeUnit unit)
                                          throws java.lang.InterruptedException
        Throws:
        java.lang.InterruptedException
      • getBackgroundCompactionCandidate

        public org.apache.cassandra.db.compaction.CompactionManager.BackgroundCompactionCandidate getBackgroundCompactionCandidate​(ColumnFamilyStore cfs)
      • performScrub

        public CompactionManager.AllSSTableOpStatus performScrub​(ColumnFamilyStore cfs,
                                                                 boolean skipCorrupted,
                                                                 boolean checkData,
                                                                 int jobs)
                                                          throws java.lang.InterruptedException,
                                                                 java.util.concurrent.ExecutionException
        Throws:
        java.lang.InterruptedException
        java.util.concurrent.ExecutionException
      • performScrub

        public CompactionManager.AllSSTableOpStatus performScrub​(ColumnFamilyStore cfs,
                                                                 boolean skipCorrupted,
                                                                 boolean checkData,
                                                                 boolean reinsertOverflowedTTL,
                                                                 int jobs)
                                                          throws java.lang.InterruptedException,
                                                                 java.util.concurrent.ExecutionException
        Throws:
        java.lang.InterruptedException
        java.util.concurrent.ExecutionException
      • performSSTableRewrite

        public CompactionManager.AllSSTableOpStatus performSSTableRewrite​(ColumnFamilyStore cfs,
                                                                          boolean excludeCurrentVersion,
                                                                          int jobs)
                                                                   throws java.lang.InterruptedException,
                                                                          java.util.concurrent.ExecutionException
        Throws:
        java.lang.InterruptedException
        java.util.concurrent.ExecutionException
      • performCleanup

        public CompactionManager.AllSSTableOpStatus performCleanup​(ColumnFamilyStore cfStore,
                                                                   int jobs)
                                                            throws java.lang.InterruptedException,
                                                                   java.util.concurrent.ExecutionException
        Throws:
        java.lang.InterruptedException
        java.util.concurrent.ExecutionException
      • relocateSSTables

        public CompactionManager.AllSSTableOpStatus relocateSSTables​(ColumnFamilyStore cfs,
                                                                     int jobs)
                                                              throws java.util.concurrent.ExecutionException,
                                                                     java.lang.InterruptedException
        Throws:
        java.util.concurrent.ExecutionException
        java.lang.InterruptedException
      • submitPendingAntiCompaction

        public com.google.common.util.concurrent.ListenableFuture<?> submitPendingAntiCompaction​(ColumnFamilyStore cfs,
                                                                                                 RangesAtEndpoint tokenRanges,
                                                                                                 Refs<SSTableReader> sstables,
                                                                                                 LifecycleTransaction txn,
                                                                                                 java.util.UUID sessionId,
                                                                                                 java.util.function.BooleanSupplier isCancelled)
        Splits the given token ranges of the given sstables into a pending repair silo
      • performAnticompaction

        public void performAnticompaction​(ColumnFamilyStore cfs,
                                          RangesAtEndpoint replicas,
                                          Refs<SSTableReader> validatedForRepair,
                                          LifecycleTransaction txn,
                                          java.util.UUID sessionID,
                                          java.util.function.BooleanSupplier isCancelled)
                                   throws java.io.IOException
        Make sure the {validatedForRepair} are marked for compaction before calling this. Caller must reference the validatedForRepair sstables (via ParentRepairSession.getActiveRepairedSSTableRefs(..)).
        Parameters:
        cfs -
        replicas - token ranges to be repaired
        validatedForRepair - SSTables containing the repaired ranges. Should be referenced before passing them.
        sessionID - the repair session we're anti-compacting for
        isCancelled - function that indicates if active anti-compaction should be canceled
        Throws:
        java.lang.InterruptedException
        java.io.IOException
      • performMaximal

        public void performMaximal​(ColumnFamilyStore cfStore,
                                   boolean splitOutput)
      • submitMaximal

        public java.util.List<java.util.concurrent.Future<?>> submitMaximal​(ColumnFamilyStore cfStore,
                                                                            int gcBefore,
                                                                            boolean splitOutput)
      • forceCompactionForTokenRange

        public void forceCompactionForTokenRange​(ColumnFamilyStore cfStore,
                                                 java.util.Collection<Range<Token>> ranges)
        Forces a major compaction of specified token ranges of the specified column family.

        The token ranges will be interpreted as closed intervals to match the closed interval defined by the first and last keys of a sstable, even though the Range class is suppossed to be half-open by definition.

        Parameters:
        cfStore - The column family store to be compacted.
        ranges - The token ranges to be compacted, interpreted as closed intervals.
      • forceUserDefinedCompaction

        public void forceUserDefinedCompaction​(java.lang.String dataFiles)
        Description copied from interface: CompactionManagerMBean
        Triggers the compaction of user specified sstables. You can specify files from various keyspaces and columnfamilies. If you do so, user defined compaction is performed several times to the groups of files in the same keyspace/columnfamily.
        Specified by:
        forceUserDefinedCompaction in interface CompactionManagerMBean
        Parameters:
        dataFiles - a comma separated list of sstable file to compact. must contain keyspace and columnfamily name in path(for 2.1+) or file name itself.
      • forceUserDefinedCleanup

        public void forceUserDefinedCleanup​(java.lang.String dataFiles)
        Description copied from interface: CompactionManagerMBean
        Triggers the cleanup of user specified sstables. You can specify files from various keyspaces and columnfamilies. If you do so, cleanup is performed each file individually
        Specified by:
        forceUserDefinedCleanup in interface CompactionManagerMBean
        Parameters:
        dataFiles - a comma separated list of sstable file to cleanup. must contain keyspace and columnfamily name in path(for 2.1+) or file name itself.
      • submitUserDefined

        public java.util.concurrent.Future<?> submitUserDefined​(ColumnFamilyStore cfs,
                                                                java.util.Collection<Descriptor> dataFiles,
                                                                int gcBefore)
      • submitValidation

        public java.util.concurrent.Future<?> submitValidation​(java.util.concurrent.Callable<java.lang.Object> validation)
      • disableAutoCompaction

        public void disableAutoCompaction()
      • needsCleanup

        public static boolean needsCleanup​(SSTableReader sstable,
                                           java.util.Collection<Range<Token>> ownedRanges)
        Determines if a cleanup would actually remove any data in this SSTable based on a set of owned ranges.
      • createWriterForAntiCompaction

        public static SSTableWriter createWriterForAntiCompaction​(ColumnFamilyStore cfs,
                                                                  java.io.File compactionFileLocation,
                                                                  int expectedBloomFilterSize,
                                                                  long repairedAt,
                                                                  java.util.UUID pendingRepair,
                                                                  boolean isTransient,
                                                                  java.util.Collection<SSTableReader> sstables,
                                                                  ILifecycleTransaction txn)
      • submitIndexBuild

        public com.google.common.util.concurrent.ListenableFuture<?> submitIndexBuild​(SecondaryIndexBuilder builder)
        Is not scheduled, because it is performing disjoint work from sstable compaction.
      • submitCacheWrite

        public java.util.concurrent.Future<?> submitCacheWrite​(AutoSavingCache.Writer writer)
      • runIndexSummaryRedistribution

        public java.util.List<SSTableReader> runIndexSummaryRedistribution​(IndexSummaryRedistribution redistribution)
                                                                    throws java.io.IOException
        Throws:
        java.io.IOException
      • getDefaultGcBefore

        public static int getDefaultGcBefore​(ColumnFamilyStore cfs,
                                             int nowInSec)
      • submitViewBuilder

        public com.google.common.util.concurrent.ListenableFuture<java.lang.Long> submitViewBuilder​(ViewBuilderTask task)
      • getActiveCompactions

        public int getActiveCompactions()
      • incrementAborted

        public void incrementAborted()
      • incrementCompactionsReduced

        public void incrementCompactionsReduced()
      • incrementSstablesDropppedFromCompactions

        public void incrementSstablesDropppedFromCompactions​(long num)
      • getTotalBytesCompacted

        public long getTotalBytesCompacted()
      • getTotalCompactionsCompleted

        public long getTotalCompactionsCompleted()
      • getPendingTasks

        public int getPendingTasks()
      • getCompletedTasks

        public long getCompletedTasks()
      • stopCompaction

        public void stopCompaction​(java.lang.String type)
        Description copied from interface: CompactionManagerMBean
        Stop all running compaction-like tasks having the provided type.
        Specified by:
        stopCompaction in interface CompactionManagerMBean
        Parameters:
        type - the type of compaction to stop. Can be one of: - COMPACTION - VALIDATION - CLEANUP - SCRUB - INDEX_BUILD
      • stopCompactionById

        public void stopCompactionById​(java.lang.String compactionId)
        Description copied from interface: CompactionManagerMBean
        Stop an individual running compaction using the compactionId.
        Specified by:
        stopCompactionById in interface CompactionManagerMBean
        Parameters:
        compactionId - Compaction ID of compaction to stop. Such IDs can be found in the transaction log files whose name starts with compaction_, located in the table transactions folder.
      • setConcurrentCompactors

        public void setConcurrentCompactors​(int value)
      • setConcurrentValidations

        public void setConcurrentValidations()
      • setConcurrentViewBuilders

        public void setConcurrentViewBuilders​(int value)
      • setCoreCompactorThreads

        public void setCoreCompactorThreads​(int number)
        Description copied from interface: CompactionManagerMBean
        Allows user to resize maximum size of the compaction thread pool.
        Specified by:
        setCoreCompactorThreads in interface CompactionManagerMBean
        Parameters:
        number - New maximum of compaction threads
      • setCoreViewBuildThreads

        public void setCoreViewBuildThreads​(int number)
        Description copied from interface: CompactionManagerMBean
        Allows user to resize maximum size of the view build thread pool.
        Specified by:
        setCoreViewBuildThreads in interface CompactionManagerMBean
        Parameters:
        number - New maximum of view build threads
      • interruptCompactionFor

        public void interruptCompactionFor​(java.lang.Iterable<TableMetadata> columnFamilies,
                                           java.util.function.Predicate<SSTableReader> sstablePredicate,
                                           boolean interruptValidation)
        Try to stop all of the compactions for given ColumnFamilies. Note that this method does not wait for all compactions to finish; you'll need to loop against isCompacting if you want that behavior.
        Parameters:
        columnFamilies - The ColumnFamilies to try to stop compaction upon.
        sstablePredicate - the sstable predicate to match on
        interruptValidation - true if validation operations for repair should also be interrupted
      • interruptCompactionForCFs

        public void interruptCompactionForCFs​(java.lang.Iterable<ColumnFamilyStore> cfss,
                                              java.util.function.Predicate<SSTableReader> sstablePredicate,
                                              boolean interruptValidation)
      • waitForCessation

        public void waitForCessation​(java.lang.Iterable<ColumnFamilyStore> cfss,
                                     java.util.function.Predicate<SSTableReader> sstablePredicate)
      • getSSTableTasks

        public java.util.List<CompactionInfo> getSSTableTasks()
      • isGlobalCompactionPaused

        public boolean isGlobalCompactionPaused()
        Return whether "global" compactions should be paused, used by ColumnFamilyStore#runWithCompactionsDisabled a global compaction is one that includes several/all tables, currently only IndexSummaryBuilder