Jobs¶
jobs
¶
Modules:
| Name | Description |
|---|---|
job |
|
jobs |
|
manager |
|
sql |
|
Functions:
| Name | Description |
|---|---|
get_parameters |
Get the job's input parameters. |
update_progress |
Update the job's progress (0-100) with an optional message. |
fail |
Mark the job as failed with an error message. |
complete |
Mark the job as completed with a result dictionary. |
get_context |
Get the context object from caller's globals (set by runpy.run_path). |
Attributes:
| Name | Type | Description |
|---|---|---|
T |
|
get_parameters
¶
update_progress
¶
fail
¶
complete
¶
get_context
¶
Get the context object from caller's globals (set by runpy.run_path).
If a type is provided, validates that the context is an instance of that type and returns it with the correct type annotation.
Source code in src/jobs/__init__.py
jobs.job
¶
Classes:
| Name | Description |
|---|---|
Job |
|
Attributes:
| Name | Type | Description |
|---|---|---|
JobStatus |
|
JobStatus
module-attribute
¶
Job
¶
Bases: ABC
Methods:
| Name | Description |
|---|---|
update_progress |
Updates the job's progress. |
start |
Starts the job. |
cancel |
Cancels the job. |
complete |
Completes the job. |
fail |
Sets the job into failed state. |
restart |
Restarts the job. |
remove |
Remove the job from the repository. |
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
The unique id of the job. |
type |
str
|
A category of the job. To allow applications to distinguish between different process types. |
name |
str
|
A unique technical name for the job. |
title |
str
|
A human-readable title which might be shown to the end-user. |
file |
Path
|
A Python file which defines the actual job logic. |
parameters |
dict[str, Any]
|
Serialized set of input parameters for the job. |
created |
datetime
|
The datetime when the job has been created (UTC). |
started_at |
datetime | None
|
The dateime when the job has been started. |
completed_at |
datetime | None
|
The datetime when the job was last updated. |
updated_at |
datetime
|
The datetime when the job was last updated. |
no_progres_timeout_seconds |
int
|
The time in seconds the job should be considered as stale when no update has been recorded. |
retention_seconds |
int
|
Time in seconds the job should be kept in database after completion before automatic deletion. |
progress |
int
|
The progress as percent value (0 - 100). |
progress_message |
str | None
|
A message (for users) which describes the current progress. |
status |
JobStatus
|
The status of the job. |
max_retries |
int
|
The number of maximum retries for the job. |
retries |
int
|
Number of current retry attempts. |
retry_timeout_seconds |
int
|
The retry timeout in seconds. |
result |
dict[str, Any] | None
|
A serialized result of the job. |
error |
str | None
|
If job has failed, this property will contain the error message. |
type
abstractmethod
property
¶
A category of the job. To allow applications to distinguish between different process types.
title
abstractmethod
property
¶
A human-readable title which might be shown to the end-user.
parameters
abstractmethod
property
¶
Serialized set of input parameters for the job.
created
abstractmethod
property
¶
The datetime when the job has been created (UTC).
started_at
abstractmethod
property
¶
The dateime when the job has been started.
completed_at
abstractmethod
property
¶
The datetime when the job was last updated.
updated_at
abstractmethod
property
¶
The datetime when the job was last updated.
no_progres_timeout_seconds
abstractmethod
property
¶
The time in seconds the job should be considered as stale when no update has been recorded. When this timeout is reached, the job process will be killed and it will be restarted.
retention_seconds
abstractmethod
property
¶
Time in seconds the job should be kept in database after completion before automatic deletion.
progress_message
abstractmethod
property
¶
A message (for users) which describes the current progress.
retry_timeout_seconds
abstractmethod
property
¶
The retry timeout in seconds.
error
abstractmethod
property
¶
If job has failed, this property will contain the error message.
update_progress
¶
Updates the job's progress.
Source code in src/jobs/job.py
start
¶
Starts the job.
Source code in src/jobs/job.py
cancel
¶
Cancels the job.
Source code in src/jobs/job.py
complete
¶
Completes the job.
Source code in src/jobs/job.py
fail
¶
Sets the job into failed state.
Source code in src/jobs/job.py
restart
¶
Restarts the job.
Source code in src/jobs/job.py
jobs.jobs
¶
Classes:
| Name | Description |
|---|---|
JobsRepositoryPort |
|
Attributes:
| Name | Type | Description |
|---|---|---|
JobOrder |
|
JobsRepositoryPort
¶
Bases: ABC
Methods:
| Name | Description |
|---|---|
find_job |
|
get_job |
|
get_jobs |
|
create |
|
get_jobs
abstractmethod
¶
create
¶
create(
*,
file: Path,
id: str | None = None,
type: str = "job",
name: str | None = None,
title: str | None = None,
parameters: dict[str, Any] | None = None,
no_progres_timeout_seconds: int = 60 * 5,
retention_seconds: int = 60 * 60 * 24,
max_retries: int = 0,
retry_timeout_seconds: int = 60
) -> Job
Source code in src/jobs/jobs.py
jobs.manager
¶
Classes:
| Name | Description |
|---|---|
JobUpdate |
|
JobsUpdate |
|
JobsManager |
|
Attributes:
| Name | Type | Description |
|---|---|---|
logger |
|
JobUpdate
dataclass
¶
JobUpdate(
id: str,
progress: int,
last_update: datetime,
progress_message: str | None,
status: JobStatus,
result: dict[str, Any] | None,
error: str | None,
)
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
|
progress |
int
|
|
last_update |
datetime
|
|
progress_message |
str | None
|
|
status |
JobStatus
|
|
result |
dict[str, Any] | None
|
|
error |
str | None
|
|
JobsUpdate
dataclass
¶
JobsUpdate(
progress: int,
last_update: datetime,
progress_message: str | None,
running_count: int,
failed: int,
runtime: timedelta,
)
Attributes:
| Name | Type | Description |
|---|---|---|
progress |
int
|
|
last_update |
datetime
|
|
progress_message |
str | None
|
|
running_count |
int
|
|
failed |
int
|
|
runtime |
timedelta
|
|
JobsManager
¶
JobsManager(
jobs: JobsRepositoryPort,
max_workers: int = 4,
recover_running: bool = False,
context: Any | None = None,
)
Methods:
| Name | Description |
|---|---|
start |
Start the executor and polling thread. |
stop |
Stop the executor gracefully. |
subscribe |
Listen to updates of a job. |
subscribe_all |
Listen to updates from multiple jobs, yielding aggregated progress. |
Attributes:
| Name | Type | Description |
|---|---|---|
jobs |
|
Source code in src/jobs/manager.py
start
¶
Start the executor and polling thread.
Source code in src/jobs/manager.py
stop
¶
Stop the executor gracefully.
Source code in src/jobs/manager.py
subscribe
¶
subscribe(
id: str, poll_interval: float = 0.5
) -> Iterator[JobUpdate]
Listen to updates of a job.
The iterator will complete when processing of the job is done.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
id
|
str
|
The id of the job to listen to. |
required |
poll_interval
|
float
|
How often to poll for updates in seconds. |
0.5
|
Source code in src/jobs/manager.py
subscribe_all
¶
subscribe_all(
ids: list[str], poll_interval: float = 0.5
) -> Iterator[JobsUpdate]
Listen to updates from multiple jobs, yielding aggregated progress.
The iterator will complete when all jobs reach a terminal status.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ids
|
list[str]
|
List of job ids to subscribe to. |
required |
poll_interval
|
float
|
How often to poll for updates in seconds. |
0.5
|
Source code in src/jobs/manager.py
131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 | |
jobs.sql
¶
Classes:
| Name | Description |
|---|---|
Base |
|
JobPE |
Persistence entity for Job. |
SQLJob |
SQL implementation of Job. |
SQLJobsRepository |
SQL implementation of JobsRepositoryPort. |
Base
¶
Bases: MappedAsDataclass, DeclarativeBase
JobPE
¶
Bases: Base
Persistence entity for Job.
Attributes:
| Name | Type | Description |
|---|---|---|
__tablename__ |
|
|
id |
Mapped[str]
|
|
type |
Mapped[str]
|
|
name |
Mapped[str]
|
|
title |
Mapped[str]
|
|
file |
Mapped[str]
|
|
created |
Mapped[datetime]
|
|
updated_at |
Mapped[datetime]
|
|
no_progress_timeout_seconds |
Mapped[int]
|
|
retention_seconds |
Mapped[int]
|
|
max_retries |
Mapped[int]
|
|
retry_timeout_seconds |
Mapped[int]
|
|
status |
Mapped[str]
|
|
parameters_json |
Mapped[str]
|
|
progress |
Mapped[int]
|
|
retries |
Mapped[int]
|
|
result_json |
Mapped[str]
|
|
started_at |
Mapped[Optional[datetime]]
|
|
completed_at |
Mapped[Optional[datetime]]
|
|
progress_message |
Mapped[Optional[str]]
|
|
error |
Mapped[Optional[str]]
|
|
id
class-attribute
instance-attribute
¶
name
class-attribute
instance-attribute
¶
updated_at
class-attribute
instance-attribute
¶
no_progress_timeout_seconds
class-attribute
instance-attribute
¶
retention_seconds
class-attribute
instance-attribute
¶
retry_timeout_seconds
class-attribute
instance-attribute
¶
status
class-attribute
instance-attribute
¶
parameters_json
class-attribute
instance-attribute
¶
progress
class-attribute
instance-attribute
¶
retries
class-attribute
instance-attribute
¶
result_json
class-attribute
instance-attribute
¶
started_at
class-attribute
instance-attribute
¶
completed_at
class-attribute
instance-attribute
¶
progress_message
class-attribute
instance-attribute
¶
error
class-attribute
instance-attribute
¶
SQLJob
¶
SQLJob(engine: Engine, job_pe: JobPE)
Bases: Job
SQL implementation of Job.
Methods:
| Name | Description |
|---|---|
remove |
|
Attributes:
| Name | Type | Description |
|---|---|---|
id |
str
|
|
type |
str
|
|
name |
str
|
|
title |
str
|
|
file |
Path
|
|
parameters |
dict[str, Any]
|
|
created |
datetime
|
|
started_at |
datetime | None
|
|
completed_at |
datetime | None
|
|
updated_at |
datetime
|
|
no_progres_timeout_seconds |
int
|
|
retention_seconds |
int
|
|
progress |
int
|
|
progress_message |
str | None
|
|
status |
JobStatus
|
|
max_retries |
int
|
|
retries |
int
|
|
retry_timeout_seconds |
int
|
|
result |
dict[str, Any] | None
|
|
error |
str | None
|
|
Source code in src/jobs/sql.py
SQLJobsRepository
¶
Bases: JobsRepositoryPort
SQL implementation of JobsRepositoryPort.
Methods:
| Name | Description |
|---|---|
create_from_settings |
|
find_job |
|
get_jobs |
|
Source code in src/jobs/sql.py
create_from_settings
staticmethod
¶
create_from_settings() -> SQLJobsRepository
get_jobs
¶
get_jobs(
type: str | None = None,
status: JobStatus | None = None,
name: str | None = None,
order: JobOrder = "created_desc",
) -> list[Job]