qsprpred.benchmarks package

Subpackages

• qsprpred.benchmarks.settings package
  • Submodules
  • qsprpred.benchmarks.settings.benchmark module
    • BenchmarkSettings
      • BenchmarkSettings.assessors
      • BenchmarkSettings.checkConsistency()
      • BenchmarkSettings.data_sources
      • BenchmarkSettings.descriptors
      • BenchmarkSettings.fromFile()
      • BenchmarkSettings.fromJSON()
      • BenchmarkSettings.models
      • BenchmarkSettings.n_replicas
      • BenchmarkSettings.name
      • BenchmarkSettings.optimizers
      • BenchmarkSettings.prep_settings
      • BenchmarkSettings.random_seed
      • BenchmarkSettings.target_props
      • BenchmarkSettings.toFile()
      • BenchmarkSettings.toJSON()
  • qsprpred.benchmarks.settings.data_prep module
    • DataPrepSettings
      • DataPrepSettings.data_filters
      • DataPrepSettings.feature_fill_value
      • DataPrepSettings.feature_filters
      • DataPrepSettings.feature_standardizer
      • DataPrepSettings.shuffle
      • DataPrepSettings.smiles_standardizer
      • DataPrepSettings.split
  • Module contents

Submodules

qsprpred.benchmarks.replica module

class qsprpred.benchmarks.replica.Replica(idx: int, name: str, data_source: DataSource, descriptors: list[qsprpred.data.descriptors.sets.DescriptorSet], target_props: list[qsprpred.tasks.TargetProperty], prep_settings: DataPrepSettings, model: QSPRModel, optimizer: HyperparameterOptimization, assessors: list[qsprpred.models.assessment.methods.ModelAssessor], random_seed: int)

Bases: JSONSerializable

Class that determines settings for a single replica of a benchmarking run.

Variables:

• idx (int) – Index of the replica. This is not an identifier, but rather a number that indicates the order of the replica in the benchmarking run.

• name (str) – Name of the replica.

• dataSource (DataSource) – Data source to use.

• descriptors (list[DescriptorSet]) – Descriptor sets to use.

• targetProps (list[TargetProperty]) – Target properties to use.

• prepSettings (DataPrepSettings) – Data preparation settings to use.

• model (QSPRModel) – Current model. Use initModel to prepare it.

• optimizer (HyperparameterOptimization) – Hyperparameter optimizer to use.

• assessors (list[ModelAssessor]) – Model assessors to use.

• randomSeed (int) – Random seed to use for all random operations withing the replica.

• ds (QSPRDataset) – Initialized data set. Only available after initData has been called.

• results (pd.DataFrame) – Results from the replica. Only available after runAssessment has been called.

addDescriptors(reload: bool = False)

Adds descriptors to the current data set. Make sure to call initData first to get it from the source.

Parameters:

reload (bool, optional) – Whether to overwrite all existing data and reinitialize from scratch. Defaults to False.

Raises:

ValueError – If the data set has not been initialized.

createReport()

Creates a report from the results of this replica.

Returns:

A pd.DataFrame with the results of this replica.

Return type:

pd.DataFrame

Raises:

ValueError – If the results have not been calculated.

classmethod fromFile(filename: str) → Any

Initialize a new instance from a JSON file.

Parameters:

filename (str) – path to the JSON file

Returns:

new instance of the class

Return type:

instance (object)

classmethod fromJSON(json: str) → Any

Reconstruct object from a JSON string.

Parameters:

json (str) – JSON string of the object

Returns:

reconstructed object

Return type:

obj (object)

getGPUs() → list[int]

Gets the GPUs to use for the model.

Returns:

List of GPU indices to use.

Return type:

list[int]

property id: str

A unique identifier for the replica.

Returns:

A unique identifier for the replica.

Return type:

str

initData(reload=False)

Initializes the data set for this replica.

Parameters:

reload (bool, optional) – Whether to overwrite all existing data and reinitialize from scratch. Defaults to False.

initModel()

Initializes the model for this replica. This includes initializing the model from the data set and optimizing the hyperparameters if an optimizer is specified.

Raises:

ValueError – If the data set has not been initialized.

prepData()

Prepares the data set for this replica.

Raises:

ValueError – If the data set has not been initialized.

property requiresGpu: bool

Whether the model requires a GPU.

Returns:

Whether the model requires a GPU.

Return type:

bool

runAssessment()

Runs the model assessment for this replica. This includes running all model assessors and saving the results.

The results are saved in the results attribute. They can be accessed by calling createReport, which combines the relevant information from the replica and the results into one pd.DataFrame.

Raises:

ValueError – If the model has not been initialized.

setGPUs(gpus: list[int])

Sets the GPUs to use for the model.

Parameters:

gpus (list[int]) – List of GPU indices to use.

toFile(filename: str) → str

Serialize object to a JSON file. This JSON file should contain all data necessary to reconstruct the object.

Parameters:

filename (str) – filename to save object to

Returns:

absolute path to the saved JSON file of the object

Return type:

filename (str)

toJSON() → str

Serialize object to a JSON string. This JSON string should

contain all data necessary to reconstruct the object.

Returns:

JSON string of the object

Return type:

json (str)

qsprpred.benchmarks.runner module

class qsprpred.benchmarks.runner.BenchmarkRunner(settings: BenchmarkSettings, data_dir: str = './data', results_file: str | None = None, parallel_generator_cpu: ParallelGenerator | None = None, parallel_generator_gpu: ParallelGenerator | None = None)

Bases: object

Class that runs benchmarking experiments as defined by BenchmarkSettings. It translates the settings into a list of Replica objects with its iterReplicas method and runs them in parallel. Each replica is processed by the runReplica method.

The report from each replica is appended to a resultsFile, which is read and returned by the run method after the runners is finished with all replicas. All outputs generated by the replicas and the BenchmarkSettings used are saved in the dataDir.

The random seed for each replica is determined in a pseudo-random way from

BenchmarkSettings.random_seed. The getSeedList method is used to generate a list of seeds from this ‘master’ seed. There are some caveats to this method (see the docstring of getSeedList).

Variables:

• settings (BenchmarkSettings) – Benchmark settings.

• nProc (int) – Number of processes to use.

• resultsFile (str) – Path to the results file.

• dataDir (str) – Path to the directory to store data.

Initialize the runner.

Parameters:

• settings (BenchmarkSettings) – Benchmark settings.

• data_dir (str, optional) – Path to the directory to store data. Defaults to “./data”. If the directory does not exist, it will be created.

• results_file (str, optional) – Path to the results file. Defaults to “{data_dir}/data/results.tsv”.

• parallel_generator_cpu (ParallelGenerator, optional) – Parallel generator for CPU replicas. Defaults to MultiprocessingPoolGenerator on all available CPUs.

• parallel_generator_gpu (ParallelGenerator, optional) – Parallel generator for GPU replicas. Defaults to None.

exception ReplicaException(replica_id: str, exception: Exception)

Bases: Exception

Custom exception for errors in a replica.

Variables:

• replicaID (int) – ID of the replica that caused the error.

• exception (Exception) – Exception that was raised.

Initialize the exception.

Parameters:

• replica_id (str) – ID of the replica that caused the error.

• exception (Exception) – Exception that was raised.

add_note()

Exception.add_note(note) – add a note to the exception

args

with_traceback()

Exception.with_traceback(tb) – set self.__traceback__ to tb and return self.

classmethod appendReportToResults(df_report: DataFrame, results_file: str)

Appends a report to the results file. This method is thread-safe.

Parameters:

• df_report (pd.DataFrame) – Report to append.

• results_file (str) – Path to the results file.

classmethod checkReplicaInResultsFile(replica: Replica, results_file: str) → bool

Checks if the replica is already present in the results file. This method is thread-safe.

Parameters:

• replica (Replica) – Replica to check.

• results_file (str) – Path to the results file.

Returns:

Whether the replica is already present in the results file.

Return type:

bool

createLocks(generator: ParallelGenerator)

Creates locks for input and output operations done in runReplica. This method should be overridden if the generator uses a different multiprocessing library or threading implementation. At the moment, this method can only provide locks for the MultiprocessingPoolGenerator.

Parameters:

generator (ParallelGenerator) – Parallel generator to use.

Returns:

Lock for data set operations. Any:

Lock for final report file operations.

Return type:

Any

Raises:

ValueError – If the generator is not a MultiprocessingPoolGenerator and lock types cannot be determined automatically.

classmethod getLoggerForReplica(replica: Replica, level: int = 10)

Returns a logger for the given replica.

Parameters:

• replica (Replica) – Replica to get the logger for.

• level (int, optional) – Log level. Defaults to logging.DEBUG.

getSeedList(seed: int | None = None) → list[int]

Get a list of seeds for the replicas from one ‘master’ randomSeed. The list of seeds is generated by sampling from the range of possible seeds (0 to 2**31-1) with the given randomSeed as the random randomSeed for the random module. This means that the list of seeds will be the same for each run of the benchmarking experiment with the same ‘master’ randomSeed. This is useful for reproducibility, but it also avoids recalculating replicas that were already calculated.

Caveat: If the randomSeed in BenchmarkSettings.randomSeed is the same, but the number of replicas is different (i.e. the settings themselves change) then this code will still generate the same seeds for experiments that might not overlap with previous experiments. Therefore, take this into account when you already calculated some replicas, but decided to change your experiment settings.

Parameters:

seed (int, optional) – ‘Master’ randomSeed. Defaults to BenchmarkSettings.randomSeed.

Returns:

list of seeds for the replicas

Return type:

list[int]

classmethod initData(replica: Replica)

Initializes the data set for this replica. This method is thread-safe.

Parameters:

replica (Replica) – Replica to initialize.

iterReplicas() → Generator[Replica, None, None]

Generator that yields Replica objects for each benchmarking run. This is done by iterating over the product of the data sources, descriptors, target properties, data preparation settings, models and optimizers as defined in the BenchmarkSettings. The random randomSeed for each replica is determined in a pseudo-random way from BenchmarkSettings.randomSeed via the getSeedList method.

Yields:

Generator[Replica, None, None] – Replica objects for each benchmarking run.

lock_data_t = <unlocked _thread.lock object>

lock_report_t = <unlocked _thread.lock object>

logLevel = 10

makeReplica(*args, **kwargs) → Replica

Returns a Replica object for the given settings. This is useful for debugging.

Returns:

Replica object.

Return type:

Replica

property nRuns: int

Returns the total number of benchmarking runs. This is the product of the number of replicas, data sources, descriptors, target properties, data preparation settings and models as defined in the BenchmarkSettings.

Returns:

Total number of benchmarking runs.

Return type:

int

processReplicas(generator: ParallelGenerator, replicas: Generator[Replica, None, None], raise_errors=False)

Processes replicas in parallel using the given ParallelGenerator. Each generated replica is run by the runReplica method, which is executed in parallel by the parallel generator according to its implementation.

Parameters:

• generator (ParallelGenerator) – Parallel generator to use.

• replicas (Generator[Replica, None, None]) – Generator that yields Replica objects.

• raise_errors (bool, optional) – Whether to raise the first encountered ReplicaException and stop the benchmarking run. Defaults to False, in which case replicas that raise an exception are skipped and errors are logged.

classmethod replicaToReport(replica: Replica) → DataFrame

Converts a replica to a report.

Parameters:

replica (Replica) – Replica to convert.

Returns:

Report from the replica.

Return type:

pd.DataFrame

run(raise_errors=False) → DataFrame

Runs the benchmarking experiments.

Parameters:

raise_errors (bool, optional) – Whether to raise the first encountered ReplicaException and stop the benchmarking run. Defaults to False, in which case replicas that raise an exception are skipped and errors are logged.

Returns:

Results from the benchmarking experiments.

Return type:

pd.DataFrame

classmethod runReplica(replica: Replica, results_file: str, lock_data: Any, lock_report: Any, gpu: int | None = None) → str | ReplicaException

Runs a single replica. This is executed in parallel by the run method. It is a classmethod so that it can be pickled and executed in parallel more easily.

Parameters:

• replica (Replica) – Replica to run.

• results_file (str) – Path to the results file.

• lock_data (Any) – Lock for data operations.

• lock_report (Any) – Lock for report operations.

Returns:

ID of the replica that was run or a ReplicaException if an error was encountered.

Return type:

str | ReplicaException

class qsprpred.benchmarks.runner.ExcThread(*args, **kwargs)

Bases: Thread

Thread that can catch exceptions from the target function.

This constructor should always be called with keyword arguments. Arguments are:

group should be None; reserved for future extension when a ThreadGroup class is implemented.

target is the callable object to be invoked by the run() method. Defaults to None, meaning nothing is called.

name is the thread name. By default, a unique name is constructed of the form “Thread-N” where N is a small decimal number.

args is a list or tuple of arguments for the target invocation. Defaults to ().

kwargs is a dictionary of keyword arguments for the target invocation. Defaults to {}.

If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread.

property daemon

A boolean value indicating whether this thread is a daemon thread.

This must be set before start() is called, otherwise RuntimeError is raised. Its initial value is inherited from the creating thread; the main thread is not a daemon thread and therefore all threads created in the main thread default to daemon = False.

The entire Python program exits when only daemon threads are left.

getName()

Return a string used for identification purposes only.

This method is deprecated, use the name attribute instead.

property ident

Thread identifier of this thread or None if it has not been started.

This is a nonzero integer. See the get_ident() function. Thread identifiers may be recycled when a thread exits and another thread is created. The identifier is available even after the thread has exited.

isDaemon()

Return whether this thread is a daemon.

This method is deprecated, use the daemon attribute instead.

is_alive()

Return whether the thread is alive.

This method returns True just before the run() method starts until just after the run() method terminates. See also the module function enumerate().

join(timeout=None)

Wait until the thread terminates.

This blocks the calling thread until the thread whose join() method is called terminates – either normally or through an unhandled exception or until the optional timeout occurs.

When the timeout argument is present and not None, it should be a floating-point number specifying a timeout for the operation in seconds (or fractions thereof). As join() always returns None, you must call is_alive() after join() to decide whether a timeout happened – if the thread is still alive, the join() call timed out.

When the timeout argument is not present or None, the operation will block until the thread terminates.

A thread can be join()ed many times.

join() raises a RuntimeError if an attempt is made to join the current thread as that would cause a deadlock. It is also an error to join() a thread before it has been started and attempts to do so raises the same exception.

property name

A string used for identification purposes only.

It has no semantics. Multiple threads may be given the same name. The initial name is set by the constructor.

property native_id

Native integral thread ID of this thread, or None if it has not been started.

This is a non-negative integer. See the get_native_id() function. This represents the Thread ID as reported by the kernel.

run()

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

setDaemon(daemonic)

Set whether this thread is a daemon.

This method is deprecated, use the .daemon property instead.

setName(name)

Set the name string for this thread.

This method is deprecated, use the name attribute instead.

start()

Start the thread’s activity.

It must be called at most once per thread object. It arranges for the object’s run() method to be invoked in a separate thread of control.

This method will raise a RuntimeError if called more than once on the same thread object.

qsprpred.benchmarks.tests module

class qsprpred.benchmarks.tests.BenchMarkTestCase(methodName='runTest')

Bases: DataSetsPathMixIn, QSPRTestCase

Test benchmarking functionality on the test data set.

Variables:

• settings (BenchmarkSettings) – Benchmark settings.

• benchmark (BenchmarkRunner) – Benchmark runner.

Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.

classmethod addClassCleanup(function, /, *args, **kwargs)

Same as addCleanup, except the cleanup items are called even if setUpClass fails (unlike tearDownClass).

addCleanup(function, /, *args, **kwargs)

Add a function, with arguments, to be called when the test is completed. Functions added are called on a LIFO basis and are called after tearDown on test failure or success.

Cleanup items are called even if setUp fails (unlike tearDown).

addTypeEqualityFunc(typeobj, function)

Add a type specific assertEqual style function to compare a type.

This method is for use by TestCase subclasses that need to register their own type equality functions to provide nicer error messages.

Parameters:

• typeobj – The data type to call this function on when both values are of the same type in assertEqual().

• function – The callable taking two arguments and an optional msg= argument that raises self.failureException with a useful error message when the two arguments are not equal.

assertAlmostEqual(first, second, places=None, msg=None, delta=None)

Fail if the two objects are unequal as determined by their difference rounded to the given number of decimal places (default 7) and comparing to zero, or by comparing that the difference between the two objects is more than the given delta.

Note that decimal places (from zero) are usually not the same as significant digits (measured from the most significant digit).

If the two objects compare equal then they will automatically compare almost equal.

assertCountEqual(first, second, msg=None)

Asserts that two iterables have the same elements, the same number of times, without regard to order.

self.assertEqual(Counter(list(first)),

Counter(list(second)))

Example:

• [0, 1, 1] and [1, 0, 1] compare equal.

• [0, 0, 1] and [0, 1] compare unequal.

assertDictEqual(d1, d2, msg=None)

assertEqual(first, second, msg=None)

Fail if the two objects are unequal as determined by the ‘==’ operator.

assertFalse(expr, msg=None)

Check that the expression is false.

assertGreater(a, b, msg=None)

Just like self.assertTrue(a > b), but with a nicer default message.

assertGreaterEqual(a, b, msg=None)

Just like self.assertTrue(a >= b), but with a nicer default message.

assertIn(member, container, msg=None)

Just like self.assertTrue(a in b), but with a nicer default message.

assertIs(expr1, expr2, msg=None)

Just like self.assertTrue(a is b), but with a nicer default message.

assertIsInstance(obj, cls, msg=None)

Same as self.assertTrue(isinstance(obj, cls)), with a nicer default message.

assertIsNone(obj, msg=None)

Same as self.assertTrue(obj is None), with a nicer default message.

assertIsNot(expr1, expr2, msg=None)

Just like self.assertTrue(a is not b), but with a nicer default message.

assertIsNotNone(obj, msg=None)

Included for symmetry with assertIsNone.

assertLess(a, b, msg=None)

Just like self.assertTrue(a < b), but with a nicer default message.

assertLessEqual(a, b, msg=None)

Just like self.assertTrue(a <= b), but with a nicer default message.

assertListEqual(list1, list2, msg=None)

A list-specific equality assertion.

Parameters:

• list1 – The first list to compare.

• list2 – The second list to compare.

• msg – Optional message to use on failure instead of a list of differences.

assertLogs(logger=None, level=None)

Fail unless a log message of level level or higher is emitted on logger_name or its children. If omitted, level defaults to INFO and logger defaults to the root logger.

This method must be used as a context manager, and will yield a recording object with two attributes: output and records. At the end of the context manager, the output attribute will be a list of the matching formatted log messages and the records attribute will be a list of the corresponding LogRecord objects.

Example:

with self.assertLogs('foo', level='INFO') as cm:
    logging.getLogger('foo').info('first message')
    logging.getLogger('foo.bar').error('second message')
self.assertEqual(cm.output, ['INFO:foo:first message',
                             'ERROR:foo.bar:second message'])

assertMultiLineEqual(first, second, msg=None)

Assert that two multi-line strings are equal.

assertNoLogs(logger=None, level=None)

Fail unless no log messages of level level or higher are emitted on logger_name or its children.

This method must be used as a context manager.

assertNotAlmostEqual(first, second, places=None, msg=None, delta=None)

Fail if the two objects are equal as determined by their difference rounded to the given number of decimal places (default 7) and comparing to zero, or by comparing that the difference between the two objects is less than the given delta.

Note that decimal places (from zero) are usually not the same as significant digits (measured from the most significant digit).

Objects that are equal automatically fail.

assertNotEqual(first, second, msg=None)

Fail if the two objects are equal as determined by the ‘!=’ operator.

assertNotIn(member, container, msg=None)

Just like self.assertTrue(a not in b), but with a nicer default message.

assertNotIsInstance(obj, cls, msg=None)

Included for symmetry with assertIsInstance.

assertNotRegex(text, unexpected_regex, msg=None)

Fail the test if the text matches the regular expression.

assertRaises(expected_exception, *args, **kwargs)

Fail unless an exception of class expected_exception is raised by the callable when invoked with specified positional and keyword arguments. If a different type of exception is raised, it will not be caught, and the test case will be deemed to have suffered an error, exactly as for an unexpected exception.

If called with the callable and arguments omitted, will return a context object used like this:

with self.assertRaises(SomeException):
    do_something()

An optional keyword argument ‘msg’ can be provided when assertRaises is used as a context object.

The context manager keeps a reference to the exception as the ‘exception’ attribute. This allows you to inspect the exception after the assertion:

with self.assertRaises(SomeException) as cm:
    do_something()
the_exception = cm.exception
self.assertEqual(the_exception.error_code, 3)

assertRaisesRegex(expected_exception, expected_regex, *args, **kwargs)

Asserts that the message in a raised exception matches a regex.

Parameters:

• expected_exception – Exception class expected to be raised.

• expected_regex – Regex (re.Pattern object or string) expected to be found in error message.

• args – Function to be called and extra positional args.

• kwargs – Extra kwargs.

• msg – Optional message used in case of failure. Can only be used when assertRaisesRegex is used as a context manager.

assertRegex(text, expected_regex, msg=None)

Fail the test unless the text matches the regular expression.

assertSequenceEqual(seq1, seq2, msg=None, seq_type=None)

An equality assertion for ordered sequences (like lists and tuples).

For the purposes of this function, a valid ordered sequence type is one which can be indexed, has a length, and has an equality operator.

Parameters:

• seq1 – The first sequence to compare.

• seq2 – The second sequence to compare.

• seq_type – The expected datatype of the sequences, or None if no datatype should be enforced.

• msg – Optional message to use on failure instead of a list of differences.

assertSetEqual(set1, set2, msg=None)

A set-specific equality assertion.

Parameters:

• set1 – The first set to compare.

• set2 – The second set to compare.

• msg – Optional message to use on failure instead of a list of differences.

assertSetEqual uses ducktyping to support different types of sets, and is optimized for sets specifically (parameters must support a difference method).

assertTrue(expr, msg=None)

Check that the expression is true.

assertTupleEqual(tuple1, tuple2, msg=None)

A tuple-specific equality assertion.

Parameters:

• tuple1 – The first tuple to compare.

• tuple2 – The second tuple to compare.

• msg – Optional message to use on failure instead of a list of differences.

assertWarns(expected_warning, *args, **kwargs)

Fail unless a warning of class warnClass is triggered by the callable when invoked with specified positional and keyword arguments. If a different type of warning is triggered, it will not be handled: depending on the other warning filtering rules in effect, it might be silenced, printed out, or raised as an exception.

If called with the callable and arguments omitted, will return a context object used like this:

with self.assertWarns(SomeWarning):
    do_something()

An optional keyword argument ‘msg’ can be provided when assertWarns is used as a context object.

The context manager keeps a reference to the first matching warning as the ‘warning’ attribute; similarly, the ‘filename’ and ‘lineno’ attributes give you information about the line of Python code from which the warning was triggered. This allows you to inspect the warning after the assertion:

with self.assertWarns(SomeWarning) as cm:
    do_something()
the_warning = cm.warning
self.assertEqual(the_warning.some_attribute, 147)

assertWarnsRegex(expected_warning, expected_regex, *args, **kwargs)

Asserts that the message in a triggered warning matches a regexp. Basic functioning is similar to assertWarns() with the addition that only warnings whose messages also match the regular expression are considered successful matches.

Parameters:

• expected_warning – Warning class expected to be triggered.

• expected_regex – Regex (re.Pattern object or string) expected to be found in error message.

• args – Function to be called and extra positional args.

• kwargs – Extra kwargs.

• msg – Optional message used in case of failure. Can only be used when assertWarnsRegex is used as a context manager.

checkRunResults(results)

checkSettings()

clearGenerated()

Remove the directories that are used for testing.

countTestCases()

createLargeMultitaskDataSet(name='QSPRDataset_multi_test', target_props=[{'name': 'HBD', 'task': <TargetTasks.MULTICLASS: 'MULTICLASS'>, 'th': [-1, 1, 2, 100]}, {'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], preparation_settings=None, random_state=42)

Create a large dataset for testing purposes.

Parameters:

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• preparation_settings (dict) – dictionary containing preparation settings

• random_state (int) – random state to use for splitting and shuffling

Returns:

a QSPRDataset object

Return type:

QSPRDataset

createLargeTestDataSet(name='QSPRDataset_test_large', target_props=[{'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], preparation_settings=None, random_state=42, n_jobs=1, chunk_size=None)

Create a large dataset for testing purposes.

Parameters:

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• random_state (int) – random state to use for splitting and shuffling

• preparation_settings (dict) – dictionary containing preparation settings

Returns:

a QSPRDataset object

Return type:

QSPRDataset

createSmallTestDataSet(name='QSPRDataset_test_small', target_props=[{'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], preparation_settings=None, random_state=42)

Create a small dataset for testing purposes.

Parameters:

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• random_state (int) – random state to use for splitting and shuffling

• preparation_settings (dict) – dictionary containing preparation settings

Returns:

a QSPRDataset object

Return type:

QSPRDataset

createTestDataSetFromFrame(df, name='QSPRDataset_test', target_props=[{'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], random_state=None, prep=None, n_jobs=1, chunk_size=None)

Create a dataset for testing purposes from the given data frame.

Parameters:

• df (pd.DataFrame) – data frame containing the dataset

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• random_state (int) – random state to use for splitting and shuffling

• prep (dict) – dictionary containing preparation settings

Returns:

a QSPRDataset object

Return type:

QSPRDataset

debug()

Run the test without collecting errors in a TestResult

defaultTestResult()

classmethod doClassCleanups()

Execute all class cleanup functions. Normally called for you after tearDownClass.

doCleanups()

Execute all cleanup functions. Normally called for you after tearDown.

classmethod enterClassContext(cm)

Same as enterContext, but class-wide.

enterContext(cm)

Enters the supplied context manager.

If successful, also adds its __exit__ method as a cleanup function and returns the result of the __enter__ method.

fail(msg=None)

Fail immediately, with the given message.

failureException

alias of AssertionError

classmethod getAllDescriptors()

Return a list of (ideally) all available descriptor sets. For now they need to be added manually to the list below.

TODO: would be nice to create the list automatically by implementing a descriptor set registry that would hold all installed descriptor sets.

Returns:

list of DescriptorCalculator objects

Return type:

list

getBigDF()

Get a large data frame for testing purposes.

Returns:

a pandas.DataFrame containing the dataset

Return type:

pd.DataFrame

classmethod getDataPrepGrid()

Return a list of many possible combinations of descriptor calculators, splits, feature standardizers, feature filters and data filters. Again, this is not exhaustive, but should cover a lot of cases.

Returns:

a generator that yields tuples of all possible combinations as stated above, each tuple is defined as: (descriptor_calculator, split, feature_standardizer, feature_filters, data_filters)

Return type:

grid

classmethod getDefaultCalculatorCombo()

Makes a list of default descriptor calculators that can be used in tests. It creates a calculator with only morgan fingerprints and rdkit descriptors, but also one with them both to test behaviour with multiple descriptor sets. Override this method if you want to test with other descriptor sets and calculator combinations.

Returns:

list of created DescriptorCalculator objects

Return type:

list

static getDefaultPrep()

Return a dictionary with default preparation settings.

classmethod getPrepCombos()

Return a list of all possible preparation combinations as generated by getDataPrepGrid as well as their names. The generated list can be used to parameterize tests with the given named combinations.

Returns:

list of `list`s of all possible combinations of preparation

Return type:

list

getSmallDF()

Get a small data frame for testing purposes.

Returns:

a pandas.DataFrame containing the dataset

Return type:

pd.DataFrame

id()

longMessage = True

maxDiff = 640

run(result=None)

setUp()

Hook method for setting up the test fixture before exercising it.

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

setUpPaths()

Create the directories that are used for testing.

shortDescription()

Returns a one-line description of the test, or None if no description has been provided.

The default implementation of this method returns the first line of the specified test method’s docstring.

skipTest(reason)

Skip this test.

subTest(msg=<object object>, **params)

Return a context manager that will return the enclosed block of code in a subtest identified by the optional message and keyword parameters. A failure in the subtest marks the test case as failed but resumes execution at the end of the enclosed block, allowing further test code to be executed.

tearDown()

Remove all files and directories that are used for testing.

classmethod tearDownClass()

Hook method for deconstructing the class fixture after running all tests in the class.

validate_split(dataset)

Check if the split has the data it should have after splitting.

class qsprpred.benchmarks.tests.BenchmarkingTest(methodName='runTest')

Bases: BenchMarkTestCase

Create an instance of the class that will use the named test method when executed. Raises a ValueError if the instance does not have a method with the specified name.

classmethod addClassCleanup(function, /, *args, **kwargs)

Same as addCleanup, except the cleanup items are called even if setUpClass fails (unlike tearDownClass).

addCleanup(function, /, *args, **kwargs)

Add a function, with arguments, to be called when the test is completed. Functions added are called on a LIFO basis and are called after tearDown on test failure or success.

Cleanup items are called even if setUp fails (unlike tearDown).

addTypeEqualityFunc(typeobj, function)

Add a type specific assertEqual style function to compare a type.

This method is for use by TestCase subclasses that need to register their own type equality functions to provide nicer error messages.

Parameters:

• typeobj – The data type to call this function on when both values are of the same type in assertEqual().

• function – The callable taking two arguments and an optional msg= argument that raises self.failureException with a useful error message when the two arguments are not equal.

assertAlmostEqual(first, second, places=None, msg=None, delta=None)

Fail if the two objects are unequal as determined by their difference rounded to the given number of decimal places (default 7) and comparing to zero, or by comparing that the difference between the two objects is more than the given delta.

Note that decimal places (from zero) are usually not the same as significant digits (measured from the most significant digit).

If the two objects compare equal then they will automatically compare almost equal.

assertCountEqual(first, second, msg=None)

Asserts that two iterables have the same elements, the same number of times, without regard to order.

self.assertEqual(Counter(list(first)),

Counter(list(second)))

Example:

• [0, 1, 1] and [1, 0, 1] compare equal.

• [0, 0, 1] and [0, 1] compare unequal.

assertDictEqual(d1, d2, msg=None)

assertEqual(first, second, msg=None)

Fail if the two objects are unequal as determined by the ‘==’ operator.

assertFalse(expr, msg=None)

Check that the expression is false.

assertGreater(a, b, msg=None)

Just like self.assertTrue(a > b), but with a nicer default message.

assertGreaterEqual(a, b, msg=None)

Just like self.assertTrue(a >= b), but with a nicer default message.

assertIn(member, container, msg=None)

Just like self.assertTrue(a in b), but with a nicer default message.

assertIs(expr1, expr2, msg=None)

Just like self.assertTrue(a is b), but with a nicer default message.

assertIsInstance(obj, cls, msg=None)

Same as self.assertTrue(isinstance(obj, cls)), with a nicer default message.

assertIsNone(obj, msg=None)

Same as self.assertTrue(obj is None), with a nicer default message.

assertIsNot(expr1, expr2, msg=None)

Just like self.assertTrue(a is not b), but with a nicer default message.

assertIsNotNone(obj, msg=None)

Included for symmetry with assertIsNone.

assertLess(a, b, msg=None)

Just like self.assertTrue(a < b), but with a nicer default message.

assertLessEqual(a, b, msg=None)

Just like self.assertTrue(a <= b), but with a nicer default message.

assertListEqual(list1, list2, msg=None)

A list-specific equality assertion.

Parameters:

• list1 – The first list to compare.

• list2 – The second list to compare.

• msg – Optional message to use on failure instead of a list of differences.

assertLogs(logger=None, level=None)

Fail unless a log message of level level or higher is emitted on logger_name or its children. If omitted, level defaults to INFO and logger defaults to the root logger.

This method must be used as a context manager, and will yield a recording object with two attributes: output and records. At the end of the context manager, the output attribute will be a list of the matching formatted log messages and the records attribute will be a list of the corresponding LogRecord objects.

Example:

with self.assertLogs('foo', level='INFO') as cm:
    logging.getLogger('foo').info('first message')
    logging.getLogger('foo.bar').error('second message')
self.assertEqual(cm.output, ['INFO:foo:first message',
                             'ERROR:foo.bar:second message'])

assertMultiLineEqual(first, second, msg=None)

Assert that two multi-line strings are equal.

assertNoLogs(logger=None, level=None)

Fail unless no log messages of level level or higher are emitted on logger_name or its children.

This method must be used as a context manager.

assertNotAlmostEqual(first, second, places=None, msg=None, delta=None)

Fail if the two objects are equal as determined by their difference rounded to the given number of decimal places (default 7) and comparing to zero, or by comparing that the difference between the two objects is less than the given delta.

Note that decimal places (from zero) are usually not the same as significant digits (measured from the most significant digit).

Objects that are equal automatically fail.

assertNotEqual(first, second, msg=None)

Fail if the two objects are equal as determined by the ‘!=’ operator.

assertNotIn(member, container, msg=None)

Just like self.assertTrue(a not in b), but with a nicer default message.

assertNotIsInstance(obj, cls, msg=None)

Included for symmetry with assertIsInstance.

assertNotRegex(text, unexpected_regex, msg=None)

Fail the test if the text matches the regular expression.

assertRaises(expected_exception, *args, **kwargs)

Fail unless an exception of class expected_exception is raised by the callable when invoked with specified positional and keyword arguments. If a different type of exception is raised, it will not be caught, and the test case will be deemed to have suffered an error, exactly as for an unexpected exception.

If called with the callable and arguments omitted, will return a context object used like this:

with self.assertRaises(SomeException):
    do_something()

An optional keyword argument ‘msg’ can be provided when assertRaises is used as a context object.

The context manager keeps a reference to the exception as the ‘exception’ attribute. This allows you to inspect the exception after the assertion:

with self.assertRaises(SomeException) as cm:
    do_something()
the_exception = cm.exception
self.assertEqual(the_exception.error_code, 3)

assertRaisesRegex(expected_exception, expected_regex, *args, **kwargs)

Asserts that the message in a raised exception matches a regex.

Parameters:

• expected_exception – Exception class expected to be raised.

• expected_regex – Regex (re.Pattern object or string) expected to be found in error message.

• args – Function to be called and extra positional args.

• kwargs – Extra kwargs.

• msg – Optional message used in case of failure. Can only be used when assertRaisesRegex is used as a context manager.

assertRegex(text, expected_regex, msg=None)

Fail the test unless the text matches the regular expression.

assertSequenceEqual(seq1, seq2, msg=None, seq_type=None)

An equality assertion for ordered sequences (like lists and tuples).

For the purposes of this function, a valid ordered sequence type is one which can be indexed, has a length, and has an equality operator.

Parameters:

• seq1 – The first sequence to compare.

• seq2 – The second sequence to compare.

• seq_type – The expected datatype of the sequences, or None if no datatype should be enforced.

• msg – Optional message to use on failure instead of a list of differences.

assertSetEqual(set1, set2, msg=None)

A set-specific equality assertion.

Parameters:

• set1 – The first set to compare.

• set2 – The second set to compare.

• msg – Optional message to use on failure instead of a list of differences.

assertSetEqual uses ducktyping to support different types of sets, and is optimized for sets specifically (parameters must support a difference method).

assertTrue(expr, msg=None)

Check that the expression is true.

assertTupleEqual(tuple1, tuple2, msg=None)

A tuple-specific equality assertion.

Parameters:

• tuple1 – The first tuple to compare.

• tuple2 – The second tuple to compare.

• msg – Optional message to use on failure instead of a list of differences.

assertWarns(expected_warning, *args, **kwargs)

Fail unless a warning of class warnClass is triggered by the callable when invoked with specified positional and keyword arguments. If a different type of warning is triggered, it will not be handled: depending on the other warning filtering rules in effect, it might be silenced, printed out, or raised as an exception.

If called with the callable and arguments omitted, will return a context object used like this:

with self.assertWarns(SomeWarning):
    do_something()

An optional keyword argument ‘msg’ can be provided when assertWarns is used as a context object.

The context manager keeps a reference to the first matching warning as the ‘warning’ attribute; similarly, the ‘filename’ and ‘lineno’ attributes give you information about the line of Python code from which the warning was triggered. This allows you to inspect the warning after the assertion:

with self.assertWarns(SomeWarning) as cm:
    do_something()
the_warning = cm.warning
self.assertEqual(the_warning.some_attribute, 147)

assertWarnsRegex(expected_warning, expected_regex, *args, **kwargs)

Asserts that the message in a triggered warning matches a regexp. Basic functioning is similar to assertWarns() with the addition that only warnings whose messages also match the regular expression are considered successful matches.

Parameters:

• expected_warning – Warning class expected to be triggered.

• expected_regex – Regex (re.Pattern object or string) expected to be found in error message.

• args – Function to be called and extra positional args.

• kwargs – Extra kwargs.

• msg – Optional message used in case of failure. Can only be used when assertWarnsRegex is used as a context manager.

checkRunResults(results)

checkSettings()

clearGenerated()

Remove the directories that are used for testing.

countTestCases()

createLargeMultitaskDataSet(name='QSPRDataset_multi_test', target_props=[{'name': 'HBD', 'task': <TargetTasks.MULTICLASS: 'MULTICLASS'>, 'th': [-1, 1, 2, 100]}, {'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], preparation_settings=None, random_state=42)

Create a large dataset for testing purposes.

Parameters:

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• preparation_settings (dict) – dictionary containing preparation settings

• random_state (int) – random state to use for splitting and shuffling

Returns:

a QSPRDataset object

Return type:

QSPRDataset

createLargeTestDataSet(name='QSPRDataset_test_large', target_props=[{'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], preparation_settings=None, random_state=42, n_jobs=1, chunk_size=None)

Create a large dataset for testing purposes.

Parameters:

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• random_state (int) – random state to use for splitting and shuffling

• preparation_settings (dict) – dictionary containing preparation settings

Returns:

a QSPRDataset object

Return type:

QSPRDataset

createSmallTestDataSet(name='QSPRDataset_test_small', target_props=[{'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], preparation_settings=None, random_state=42)

Create a small dataset for testing purposes.

Parameters:

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• random_state (int) – random state to use for splitting and shuffling

• preparation_settings (dict) – dictionary containing preparation settings

Returns:

a QSPRDataset object

Return type:

QSPRDataset

createTestDataSetFromFrame(df, name='QSPRDataset_test', target_props=[{'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], random_state=None, prep=None, n_jobs=1, chunk_size=None)

Create a dataset for testing purposes from the given data frame.

Parameters:

• df (pd.DataFrame) – data frame containing the dataset

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• random_state (int) – random state to use for splitting and shuffling

• prep (dict) – dictionary containing preparation settings

Returns:

a QSPRDataset object

Return type:

QSPRDataset

debug()

Run the test without collecting errors in a TestResult

defaultTestResult()

classmethod doClassCleanups()

Execute all class cleanup functions. Normally called for you after tearDownClass.

doCleanups()

Execute all cleanup functions. Normally called for you after tearDown.

classmethod enterClassContext(cm)

Same as enterContext, but class-wide.

enterContext(cm)

Enters the supplied context manager.

If successful, also adds its __exit__ method as a cleanup function and returns the result of the __enter__ method.

fail(msg=None)

Fail immediately, with the given message.

failureException

alias of AssertionError

classmethod getAllDescriptors()

Return a list of (ideally) all available descriptor sets. For now they need to be added manually to the list below.

TODO: would be nice to create the list automatically by implementing a descriptor set registry that would hold all installed descriptor sets.

Returns:

list of DescriptorCalculator objects

Return type:

list

getBigDF()

Get a large data frame for testing purposes.

Returns:

a pandas.DataFrame containing the dataset

Return type:

pd.DataFrame

classmethod getDataPrepGrid()

Return a list of many possible combinations of descriptor calculators, splits, feature standardizers, feature filters and data filters. Again, this is not exhaustive, but should cover a lot of cases.

Returns:

a generator that yields tuples of all possible combinations as stated above, each tuple is defined as: (descriptor_calculator, split, feature_standardizer, feature_filters, data_filters)

Return type:

grid

classmethod getDefaultCalculatorCombo()

Makes a list of default descriptor calculators that can be used in tests. It creates a calculator with only morgan fingerprints and rdkit descriptors, but also one with them both to test behaviour with multiple descriptor sets. Override this method if you want to test with other descriptor sets and calculator combinations.

Returns:

list of created DescriptorCalculator objects

Return type:

list

static getDefaultPrep()

Return a dictionary with default preparation settings.

classmethod getPrepCombos()

Return a list of all possible preparation combinations as generated by getDataPrepGrid as well as their names. The generated list can be used to parameterize tests with the given named combinations.

Returns:

list of `list`s of all possible combinations of preparation

Return type:

list

getSmallDF()

Get a small data frame for testing purposes.

Returns:

a pandas.DataFrame containing the dataset

Return type:

pd.DataFrame

id()

longMessage = True

maxDiff = 640

run(result=None)

setUp()

Hook method for setting up the test fixture before exercising it.

classmethod setUpClass()

Hook method for setting up class fixture before running tests in the class.

setUpPaths()

Create the directories that are used for testing.

shortDescription()

Returns a one-line description of the test, or None if no description has been provided.

The default implementation of this method returns the first line of the specified test method’s docstring.

skipTest(reason)

Skip this test.

subTest(msg=<object object>, **params)

Return a context manager that will return the enclosed block of code in a subtest identified by the optional message and keyword parameters. A failure in the subtest marks the test case as failed but resumes execution at the end of the enclosed block, allowing further test code to be executed.

tearDown()

Remove all files and directories that are used for testing.

classmethod tearDownClass()

Hook method for deconstructing the class fixture after running all tests in the class.

testMultiTaskCLS()

Run the test benchmark.

testMultiTaskREG()

testSingleTaskCLS()

Run single task tests for classification.

testSingleTaskREG()

validate_split(dataset)

Check if the split has the data it should have after splitting.

class qsprpred.benchmarks.tests.DataSourceTesting(name)

Bases: DataSetsPathMixIn, DataSource

Data source for testing purposes. Simply prepares the default data set from`DataSetsPathMixIn`.

clearGenerated()

Remove the directories that are used for testing.

createLargeMultitaskDataSet(name='QSPRDataset_multi_test', target_props=[{'name': 'HBD', 'task': <TargetTasks.MULTICLASS: 'MULTICLASS'>, 'th': [-1, 1, 2, 100]}, {'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], preparation_settings=None, random_state=42)

Create a large dataset for testing purposes.

Parameters:

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• preparation_settings (dict) – dictionary containing preparation settings

• random_state (int) – random state to use for splitting and shuffling

Returns:

a QSPRDataset object

Return type:

QSPRDataset

createLargeTestDataSet(name='QSPRDataset_test_large', target_props=[{'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], preparation_settings=None, random_state=42, n_jobs=1, chunk_size=None)

Create a large dataset for testing purposes.

Parameters:

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• random_state (int) – random state to use for splitting and shuffling

• preparation_settings (dict) – dictionary containing preparation settings

Returns:

a QSPRDataset object

Return type:

QSPRDataset

createSmallTestDataSet(name='QSPRDataset_test_small', target_props=[{'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], preparation_settings=None, random_state=42)

Create a small dataset for testing purposes.

Parameters:

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• random_state (int) – random state to use for splitting and shuffling

• preparation_settings (dict) – dictionary containing preparation settings

Returns:

a QSPRDataset object

Return type:

QSPRDataset

createTestDataSetFromFrame(df, name='QSPRDataset_test', target_props=[{'name': 'CL', 'task': <TargetTasks.REGRESSION: 'REGRESSION'>}], random_state=None, prep=None, n_jobs=1, chunk_size=None)

Create a dataset for testing purposes from the given data frame.

Parameters:

• df (pd.DataFrame) – data frame containing the dataset

• name (str) – name of the dataset

• target_props (List of dicts or TargetProperty) – list of target properties

• random_state (int) – random state to use for splitting and shuffling

• prep (dict) – dictionary containing preparation settings

Returns:

a QSPRDataset object

Return type:

QSPRDataset

classmethod getAllDescriptors()

Return a list of (ideally) all available descriptor sets. For now they need to be added manually to the list below.

TODO: would be nice to create the list automatically by implementing a descriptor set registry that would hold all installed descriptor sets.

Returns:

list of DescriptorCalculator objects

Return type:

list

getBigDF()

Get a large data frame for testing purposes.

Returns:

a pandas.DataFrame containing the dataset

Return type:

pd.DataFrame

getData(name: str | None = None, **kwargs) → MoleculeTable

classmethod getDataPrepGrid()

Return a list of many possible combinations of descriptor calculators, splits, feature standardizers, feature filters and data filters. Again, this is not exhaustive, but should cover a lot of cases.

Returns:

a generator that yields tuples of all possible combinations as stated above, each tuple is defined as: (descriptor_calculator, split, feature_standardizer, feature_filters, data_filters)

Return type:

grid

getDataSet(target_props: list[qsprpred.tasks.TargetProperty | dict], name: str | None = None, **kwargs) → QSPRDataset

classmethod getDefaultCalculatorCombo()

Makes a list of default descriptor calculators that can be used in tests. It creates a calculator with only morgan fingerprints and rdkit descriptors, but also one with them both to test behaviour with multiple descriptor sets. Override this method if you want to test with other descriptor sets and calculator combinations.

Returns:

list of created DescriptorCalculator objects

Return type:

list

static getDefaultPrep()

Return a dictionary with default preparation settings.

classmethod getPrepCombos()

Return a list of all possible preparation combinations as generated by getDataPrepGrid as well as their names. The generated list can be used to parameterize tests with the given named combinations.

Returns:

list of `list`s of all possible combinations of preparation

Return type:

list

getSmallDF()

Get a small data frame for testing purposes.

Returns:

a pandas.DataFrame containing the dataset

Return type:

pd.DataFrame

setUpPaths()

Create the directories that are used for testing.

tearDown()

Remove all files and directories that are used for testing.

validate_split(dataset)

Check if the split has the data it should have after splitting.

Module contents